Jul 15 07:58:44.605602 ip-10-0-128-37 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 07:58:44.605617 ip-10-0-128-37 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 07:58:44.605627 ip-10-0-128-37 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 07:58:44.605945 ip-10-0-128-37 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 07:58:54.855411 ip-10-0-128-37 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 07:58:54.855430 ip-10-0-128-37 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b45e7e0716684c559e61152f5483ee62 -- Jul 15 08:00:42.363704 ip-10-0-128-37 systemd[1]: Starting Kubernetes Kubelet... Jul 15 08:00:42.869632 ip-10-0-128-37 systemd[1]: Started Kubernetes Kubelet. Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.801040 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804499 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804509 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804513 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804516 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804519 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804522 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804525 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:42.875149 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804527 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804530 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804533 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804535 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804538 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804540 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804543 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804546 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804548 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804552 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804561 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804566 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804570 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804573 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804576 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804578 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804581 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804584 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804586 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:42.876849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804589 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804591 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804594 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804598 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804600 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804603 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804605 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804608 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804610 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804613 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804615 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804618 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804620 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804624 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804626 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804630 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804632 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804636 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804638 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804642 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:42.877880 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804645 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804647 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804650 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804652 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804655 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804658 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804661 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804663 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804666 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804668 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804671 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804673 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804676 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804679 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804681 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804684 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804686 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804689 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804691 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804694 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:42.878877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804696 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804698 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804701 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804703 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804707 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804709 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804712 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804714 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804717 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804719 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804722 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804724 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804729 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804733 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804737 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804739 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804743 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804746 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804748 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:42.879520 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804751 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804754 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804756 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804759 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804761 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804764 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804766 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804769 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804771 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804774 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804776 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804779 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804781 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.804784 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805500 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805507 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805511 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805514 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805516 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805519 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:42.880962 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805523 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805526 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805529 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805532 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805535 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805537 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805540 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805543 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805546 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805549 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805552 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805556 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805558 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805561 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805564 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805566 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805569 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805572 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805574 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805577 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805579 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:42.882233 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805582 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805585 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805587 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805590 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805594 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805598 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805601 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805604 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805608 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805611 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805613 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805617 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805619 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805622 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805625 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805627 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805630 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805632 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805635 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:42.884472 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805637 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805640 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805642 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805645 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805648 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805650 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805653 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805655 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805658 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805660 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805663 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805667 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805670 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805673 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805676 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805679 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805682 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805685 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805689 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805692 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:42.885340 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805694 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805697 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805700 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805703 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805706 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805710 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805713 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805716 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805718 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805721 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805723 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805726 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805729 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805731 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805735 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805739 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805741 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805744 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805747 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:42.887040 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805749 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805752 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805754 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805757 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805759 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805762 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805765 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805767 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805770 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805772 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805775 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805778 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805780 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.805783 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806118 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806122 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806125 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806128 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806131 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806134 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:42.887681 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806137 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806140 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806142 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806145 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806150 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806153 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806155 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806158 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806160 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806163 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806165 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806168 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806171 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806173 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806176 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806179 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806181 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806184 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806187 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806189 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:42.888757 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806192 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806194 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806197 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806200 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806203 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806206 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806209 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806211 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806214 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806218 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806222 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806225 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806228 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806231 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806234 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806236 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806240 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806243 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806246 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:42.889812 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806248 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806251 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806254 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806256 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806259 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806262 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806265 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806268 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806270 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806273 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806275 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806278 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806280 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806283 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806285 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806288 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806290 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806293 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806295 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806298 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:42.892027 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806301 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806303 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806306 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806308 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806311 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806320 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806323 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806325 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806328 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806333 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806336 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806339 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806341 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806343 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806346 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806349 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806352 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806355 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806359 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:42.893074 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806362 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806365 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806368 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806371 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806373 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806376 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806379 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806381 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806384 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806386 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806389 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806391 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806394 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806396 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.806399 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806480 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806486 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806492 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806496 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806506 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806510 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 08:00:42.894354 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806514 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806519 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806522 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806525 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806528 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806532 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806534 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806538 2567 flags.go:64] FLAG: --cgroup-root="" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806541 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806544 2567 flags.go:64] FLAG: --client-ca-file="" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806547 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806550 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806557 2567 flags.go:64] FLAG: --cluster-domain="" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806559 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806562 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806565 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806570 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806574 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806577 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806580 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806583 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806586 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806589 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806591 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 08:00:42.895390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806595 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806597 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806601 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806604 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806607 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806610 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806613 2567 flags.go:64] FLAG: --enable-server="true" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806621 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806628 2567 flags.go:64] FLAG: --event-burst="100" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806632 2567 flags.go:64] FLAG: --event-qps="50" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806635 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806638 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806641 2567 flags.go:64] FLAG: --eviction-hard="" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806645 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806647 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806650 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806653 2567 flags.go:64] FLAG: --eviction-soft="" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806656 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806659 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806662 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806664 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806667 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806670 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806672 2567 flags.go:64] FLAG: --feature-gates="" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806677 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 08:00:42.903046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806680 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806683 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806686 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806689 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806692 2567 flags.go:64] FLAG: --help="false" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806694 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-128-37.ec2.internal" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806697 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806701 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806703 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806707 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806710 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806713 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806716 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806718 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806721 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806725 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806729 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806732 2567 flags.go:64] FLAG: --kube-reserved="" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806735 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806737 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806740 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806743 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806746 2567 flags.go:64] FLAG: --lock-file="" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806748 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 08:00:42.903946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806751 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806754 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806759 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806762 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806764 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806767 2567 flags.go:64] FLAG: --logging-format="text" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806770 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806774 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806777 2567 flags.go:64] FLAG: --manifest-url="" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806780 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806784 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806787 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806791 2567 flags.go:64] FLAG: --max-pods="110" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806793 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806796 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806799 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806802 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806805 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806808 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806810 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806818 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806820 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806823 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 08:00:42.905489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806826 2567 flags.go:64] FLAG: --pod-cidr="" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806831 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806836 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806838 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806841 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806844 2567 flags.go:64] FLAG: --port="10250" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806847 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806850 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cbbab48bfd2fdd66" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806853 2567 flags.go:64] FLAG: --qos-reserved="" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806855 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806859 2567 flags.go:64] FLAG: --register-node="true" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806861 2567 flags.go:64] FLAG: --register-with-taints="" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806865 2567 flags.go:64] FLAG: --registry-burst="10" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806867 2567 flags.go:64] FLAG: --registry-qps="5" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806870 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806873 2567 flags.go:64] FLAG: --reserved-memory="" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806878 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806881 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806884 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806887 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806889 2567 flags.go:64] FLAG: --runonce="false" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806892 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806895 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806898 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806901 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 08:00:42.906182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806903 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806906 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806909 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806912 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806915 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806918 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806920 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806923 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806926 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806934 2567 flags.go:64] FLAG: --system-cgroups="" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806937 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806942 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806944 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806947 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806955 2567 flags.go:64] FLAG: --tls-min-version="" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806958 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806961 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806964 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806967 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806969 2567 flags.go:64] FLAG: --v="2" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806973 2567 flags.go:64] FLAG: --version="false" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806977 2567 flags.go:64] FLAG: --vmodule="" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806985 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.806990 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807095 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:43.053312 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807098 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807101 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807104 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807107 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807109 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807112 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807115 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807119 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807122 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807125 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807128 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807130 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807133 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807135 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807138 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807140 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807144 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807147 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807150 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807152 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:43.069156 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807155 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807157 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807160 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807163 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807165 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807168 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807170 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807173 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807176 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807178 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807182 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807185 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807187 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807190 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807193 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807195 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807198 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807200 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807203 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807205 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:43.070545 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807208 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807210 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807213 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807215 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807218 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807220 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807223 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807225 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807229 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807233 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807236 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807239 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807242 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807245 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807247 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807250 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807254 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807257 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807260 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:43.071169 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807263 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807266 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807269 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807273 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807276 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807278 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807281 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807284 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807286 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807289 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807291 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807294 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807296 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807299 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807302 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807304 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807307 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807309 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807311 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807314 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:43.071949 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807317 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807320 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807323 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807325 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807328 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807331 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807333 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807336 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807339 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807341 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807344 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807346 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807349 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807352 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807354 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807359 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807361 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807364 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:43.072603 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.807366 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.807372 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.813935 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.813953 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.813970 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.813982 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814054 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814062 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814068 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814073 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814079 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814083 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814087 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814091 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814095 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:43.073313 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814100 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814103 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814108 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814112 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814116 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814121 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814126 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814130 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814135 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814140 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814145 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814149 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814153 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814157 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814161 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814165 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814170 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814174 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814178 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:43.073802 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814182 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814186 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814190 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814194 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814198 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814202 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814205 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814209 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814213 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814217 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814221 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814225 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814228 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814233 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814237 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814240 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814244 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814248 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814254 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814258 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:43.074633 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814261 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814265 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814269 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814273 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814277 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814280 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814284 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814287 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814291 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814295 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814299 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814302 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814307 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814310 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814315 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814319 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814322 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814325 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814329 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814333 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:43.075264 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814339 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814343 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814346 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814350 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814354 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814358 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814361 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814366 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814369 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814373 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814378 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814382 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814386 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814390 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814394 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814397 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814401 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814405 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814409 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814412 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:43.076162 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814416 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814419 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814424 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814427 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814431 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814455 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814460 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814465 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814470 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814473 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814478 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.814484 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814638 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814647 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814651 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:43.076759 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814655 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814659 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814662 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814666 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814670 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814673 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814677 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814683 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814687 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814691 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814694 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814698 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814702 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814706 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814710 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814715 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814719 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814723 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814727 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814731 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:43.077385 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814734 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814738 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814742 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814745 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814749 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814754 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814758 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814761 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814765 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814769 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814774 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814778 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814781 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814785 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814789 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814793 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814797 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814800 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814804 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814809 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:43.077969 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814813 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814817 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814820 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814824 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814828 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814832 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814838 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814843 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814847 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814851 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814855 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814859 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814864 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814868 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814872 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814875 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814879 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814883 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814888 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:43.232102 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814892 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814895 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814899 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814904 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814910 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814915 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814919 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814923 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814927 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814932 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814935 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814940 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814946 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814950 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814954 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814958 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814962 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814967 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814971 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814975 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:43.233253 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814979 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814983 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814986 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814990 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814994 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.814998 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815001 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815005 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815009 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815013 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815017 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815020 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815026 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815030 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815033 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815039 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:43.235991 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:42.815043 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.815050 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.815175 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.818227 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.819500 2567 server.go:1019] "Starting client certificate rotation" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.819604 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.819644 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.823080 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.829162 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.840153 2567 log.go:25] "Validated CRI v1 runtime API" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.842029 2567 log.go:25] "Validated CRI v1 image API" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.844113 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 08:00:43.236616 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.846392 2567 fs.go:135] Filesystem UUIDs: map[6faf61eb-2326-4bf0-9d4f-f92179f3970e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a25b1ceb-2182-4494-83e2-7f08afb3520b:/dev/nvme0n1p3] Jul 15 08:00:43.237266 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.846405 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 08:00:43.237266 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.848427 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.850319 2567 manager.go:216] Machine: {Timestamp:2026-07-15 08:00:42.849852094 +0000 UTC m=+0.365648919 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100814 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec209f5ad181f3f98b45f14a1086a4d4 SystemUUID:ec209f5a-d181-f3f9-8b45-f14a1086a4d4 BootID:b45e7e07-1668-4c55-9e61-152f5483ee62 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2a:31:b8:89:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2a:31:b8:89:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:1d:e8:41:ab:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.850415 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.850495 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.853614 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.853636 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.854347 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.854356 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.854367 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.854993 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.856176 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859450 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859576 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859598 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859607 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859621 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859629 2567 kubelet.go:398] "Adding apiserver pod source" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859637 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 08:00:43.237707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859870 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.859883 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.863346 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.864937 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.864964 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865623 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865638 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865643 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865650 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865661 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865666 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865674 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865679 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865685 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865691 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865700 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865705 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865719 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.865724 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.868871 2567 server.go:1268] "Started kubelet" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.868937 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.869338 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.869644 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.870745 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.871272 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-596q2" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.874160 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.874734 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.874972 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.875306 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.877857 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.878939 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-596q2" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.879731 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.879740 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.880018 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.880424 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.880456 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.880431 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.880586 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.880592 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.880965 2567 factory.go:55] Registering systemd factory Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.880979 2567 factory.go:223] Registration of the systemd container factory successfully Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.881260 2567 factory.go:153] Registering CRI-O factory Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.881270 2567 factory.go:223] Registration of the crio container factory successfully Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.881318 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.881339 2567 factory.go:103] Registering Raw factory Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.881354 2567 manager.go:1196] Started watching for new ooms in manager Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.882012 2567 manager.go:349] Starting recovery of all containers Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.893941 2567 manager.go:354] Recovery completed Jul 15 08:00:43.238303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.898337 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.900146 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.900637 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.900653 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.899157 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-37.ec2.internal.18c267b81e1ab479 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-37.ec2.internal,UID:ip-10-0-128-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-37.ec2.internal,},FirstTimestamp:2026-07-15 08:00:42.868847737 +0000 UTC m=+0.384644562,LastTimestamp:2026-07-15 08:00:42.868847737 +0000 UTC m=+0.384644562,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-37.ec2.internal,}" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.901182 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.901187 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.901206 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.901230 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.902071 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.902096 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.902113 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.902514 2567 policy_none.go:49] "None policy: Start" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.902537 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.902550 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.903574 2567 policy_none.go:47] "Start" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.909807 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.912798 2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-37.ec2.internal\" not found" node="ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.948130 2567 manager.go:337] "Starting Device Plugin manager" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.948155 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.948164 2567 server.go:87] "Starting device plugin registration server" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.948339 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.948351 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.948454 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.948560 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:42.948570 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.948945 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:42.948980 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.026295 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.027453 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.027492 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.027514 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.027553 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.029974 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.048771 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.049525 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.049548 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.049558 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.049578 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.057699 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.057715 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-37.ec2.internal\": node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.239812 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.075363 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.127872 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal"] Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.127918 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.129203 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.129225 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.129239 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.130534 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.130686 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.130716 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.135272 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.135290 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.135299 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.135369 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.135395 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.135406 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.136503 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.136522 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.137073 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.137101 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.137113 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.169922 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-37.ec2.internal\" not found" node="ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.174054 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-37.ec2.internal\" not found" node="ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.176084 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.182267 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9545b0e517d36e2a7fb3813f2cac8af0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-37.ec2.internal\" (UID: \"9545b0e517d36e2a7fb3813f2cac8af0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.182293 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf158e8b3088cd152953b639a569c880-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"bf158e8b3088cd152953b639a569c880\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.241162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.182311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf158e8b3088cd152953b639a569c880-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"bf158e8b3088cd152953b639a569c880\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.381497 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.276174 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.381497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.282944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9545b0e517d36e2a7fb3813f2cac8af0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-37.ec2.internal\" (UID: \"9545b0e517d36e2a7fb3813f2cac8af0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.381497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.282966 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf158e8b3088cd152953b639a569c880-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"bf158e8b3088cd152953b639a569c880\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.381497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.282983 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf158e8b3088cd152953b639a569c880-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"bf158e8b3088cd152953b639a569c880\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.381497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.283019 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf158e8b3088cd152953b639a569c880-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"bf158e8b3088cd152953b639a569c880\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.381497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.283048 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9545b0e517d36e2a7fb3813f2cac8af0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-37.ec2.internal\" (UID: \"9545b0e517d36e2a7fb3813f2cac8af0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.381497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.283065 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf158e8b3088cd152953b639a569c880-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"bf158e8b3088cd152953b639a569c880\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.381497 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.376666 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.568508 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.477851 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.568508 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.477897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.568508 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.477966 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:43.674695 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.577902 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.704361 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.704334 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.805279 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.805256 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:43.819487 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.819462 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 08:00:43.819607 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.819590 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 08:00:43.819648 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.819620 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 08:00:43.883970 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.883918 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 07:55:43 +0000 UTC" deadline="2028-01-14 05:32:32.758055321 +0000 UTC" Jul 15 08:00:43.883970 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.883962 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13149h31m48.874098029s" Jul 15 08:00:43.891806 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:43.891781 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:43.906139 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:43.906117 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:44.008182 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:44.008134 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found" Jul 15 08:00:44.042715 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.042697 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:44.082733 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.082709 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" Jul 15 08:00:44.101184 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.101163 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 08:00:44.102096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.102083 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" Jul 15 08:00:44.110636 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.110621 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 08:00:44.418387 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:44.418359 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf158e8b3088cd152953b639a569c880.slice/crio-8286f827de82ab66e2f92328971f2c2d2cb96a22973cd02066c2e520ecfb83d9 WatchSource:0}: Error finding container 8286f827de82ab66e2f92328971f2c2d2cb96a22973cd02066c2e520ecfb83d9: Status 404 returned error can't find the container with id 8286f827de82ab66e2f92328971f2c2d2cb96a22973cd02066c2e520ecfb83d9 Jul 15 08:00:44.422389 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.422370 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:00:44.863952 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.863921 2567 apiserver.go:52] "Watching apiserver" Jul 15 08:00:44.874321 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.874292 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 08:00:44.874770 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.874742 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal","openshift-multus/multus-additional-cni-plugins-47mlt","openshift-multus/multus-hk8lc","kube-system/global-pull-secret-syncer-nlzkz","openshift-image-registry/node-ca-clz7b","openshift-multus/network-metrics-daemon-f6cf9","openshift-network-diagnostics/network-check-target-7tfd8","openshift-network-operator/iptables-alerter-57h8c","openshift-ovn-kubernetes/ovnkube-node-bnbv5","kube-system/konnectivity-agent-fz4s9","kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k","openshift-cluster-node-tuning-operator/tuned-bvrzh","openshift-dns/node-resolver-txzrm"] Jul 15 08:00:44.876239 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.876220 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:44.876344 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:44.876303 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:00:44.877560 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.877516 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.878695 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.878670 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.880024 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.879961 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:44.880130 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.880021 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r5k5p\"" Jul 15 08:00:44.880130 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:44.880027 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:00:44.880130 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.880065 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 08:00:44.880308 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.880142 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:00:44.881403 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.880598 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 08:00:44.881403 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.881179 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:44.881950 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.881927 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8rllf\"" Jul 15 08:00:44.882304 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.882287 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 08:00:44.882459 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.882403 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 08:00:44.882563 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.882494 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 08:00:44.882666 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.882620 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 08:00:44.882832 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.882811 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:44.882927 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:44.882881 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:00:44.883257 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.883237 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tlf9\"" Jul 15 08:00:44.883454 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.883429 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 08:00:44.884029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.884013 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 08:00:44.884637 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.884160 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:44.884637 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.884256 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 08:00:44.885549 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.885519 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:44.886392 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.886372 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 08:00:44.886780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.886758 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tqfd4\"" Jul 15 08:00:44.886780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.886778 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 08:00:44.886942 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.886818 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 08:00:44.887148 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.887130 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 08:00:44.887344 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.887328 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 08:00:44.887423 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.887375 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 08:00:44.887658 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.887637 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:44.887998 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.887876 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 08:00:44.888110 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.888029 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 08:00:44.888171 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.888055 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p8n4t\"" Jul 15 08:00:44.888895 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.888877 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:00:44.890043 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.890022 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 08:00:44.890122 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.890089 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 08:00:44.890404 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.890382 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hcvx2\"" Jul 15 08:00:44.891058 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.891037 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:44.891618 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.891596 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 08:00:44.891718 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.891665 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 08:00:44.891777 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.891758 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tct8x\"" Jul 15 08:00:44.893066 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.893048 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:44.893896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.893707 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8hg6h\"" Jul 15 08:00:44.893896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.893725 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:00:44.893896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.893740 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 08:00:44.893896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.893738 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 08:00:44.896579 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.896394 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 08:00:44.896685 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.896635 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 08:00:44.896778 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.896762 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 08:00:44.896841 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.896777 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkszc\"" Jul 15 08:00:44.898884 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.898852 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 08:00:44.918673 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918656 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8a42fee-4675-42df-8f1a-71cb407b62ad-kubelet-config\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:44.918770 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918684 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8a42fee-4675-42df-8f1a-71cb407b62ad-dbus\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:44.918770 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-lib-modules\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.918770 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918732 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-var-lib-kubelet\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.918770 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918756 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/708eea1b-63a0-46a1-b87d-42f8c7b3f318-tmp\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.918963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-conf-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.918963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918808 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d28ck\" (UniqueName: \"kubernetes.io/projected/451a8c3d-179c-4a38-a3c3-50c96043ebf1-kube-api-access-d28ck\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:44.918963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918856 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-tuned\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.918963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-socket-dir-parent\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.918963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918917 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-hostroot\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.918968 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g946\" (UniqueName: \"kubernetes.io/projected/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-kube-api-access-9g946\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysctl-d\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-system-cni-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-cnibin\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-cni-bin\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-modprobe-d\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919154 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysconfig\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919184 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysctl-conf\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919207 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-host\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-cni-multus\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919259 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5hb\" (UniqueName: \"kubernetes.io/projected/708eea1b-63a0-46a1-b87d-42f8c7b3f318-kube-api-access-dw5hb\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919283 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-cni-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919307 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-cni-binary-copy\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-kubernetes\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-daemon-config\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919381 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-os-release\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919430 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-netns\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-multus-certs\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919500 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-etc-kubernetes\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919574 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/451a8c3d-179c-4a38-a3c3-50c96043ebf1-host\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919635 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/451a8c3d-179c-4a38-a3c3-50c96043ebf1-serviceca\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-systemd\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919705 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-run\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.919762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919731 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-sys\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:44.920524 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-k8s-cni-cncf-io\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.920524 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.919803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-kubelet\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:44.922300 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.922281 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j9vsk" Jul 15 08:00:44.931256 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.931238 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j9vsk" Jul 15 08:00:44.982149 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:44.982125 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 08:00:45.019970 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.019942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-tuned\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.020067 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.019977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g946\" (UniqueName: \"kubernetes.io/projected/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-kube-api-access-9g946\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.020067 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020008 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.020176 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-device-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.020220 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020185 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-node-log\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.020220 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020213 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-ovnkube-script-lib\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.020301 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-cnibin\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.020301 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-cni-bin\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.020385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-hosts-file\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.020385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020314 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 08:00:45.020385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-registration-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.020385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020348 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-cnibin\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.020385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020350 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-etc-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020394 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-ovn\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020402 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-cni-bin\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020424 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswf2\" (UniqueName: \"kubernetes.io/projected/bc482a89-5896-486c-ac28-9569d5b79c44-kube-api-access-rswf2\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysconfig\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020498 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysctl-conf\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020521 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysconfig\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020567 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-host\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020616 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-host\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-cni-multus\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020641 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysctl-conf\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020653 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-os-release\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.020663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020657 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-cni-multus\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-iptables-alerter-script\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020705 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-log-socket\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020760 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020789 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-system-cni-dir\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-tmp-dir\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020833 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-var-lib-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cnibin\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-systemd-units\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc482a89-5896-486c-ac28-9569d5b79c44-ovn-node-metrics-cert\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020929 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-os-release\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.020974 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-netns\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021021 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9z9k\" (UniqueName: \"kubernetes.io/projected/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-kube-api-access-c9z9k\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021048 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-sys-fs\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8qj\" (UniqueName: \"kubernetes.io/projected/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-kube-api-access-7b8qj\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021048 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-os-release\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.021238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-kubelet\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-run\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4t84\" (UniqueName: \"kubernetes.io/projected/98640408-aff8-4f6d-8928-15afdf0e3ae3-kube-api-access-d4t84\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021080 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-netns\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/708eea1b-63a0-46a1-b87d-42f8c7b3f318-tmp\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021203 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d28ck\" (UniqueName: \"kubernetes.io/projected/451a8c3d-179c-4a38-a3c3-50c96043ebf1-kube-api-access-d28ck\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021234 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-run\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-slash\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-env-overrides\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021365 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-socket-dir-parent\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-hostroot\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfsxj\" (UniqueName: \"kubernetes.io/projected/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-kube-api-access-kfsxj\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021482 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-socket-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysctl-d\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.022096 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021557 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-system-cni-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-systemd\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021605 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-socket-dir-parent\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021637 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-hostroot\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021656 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-ovnkube-config\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-modprobe-d\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-sysctl-d\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021729 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-system-cni-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021743 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c517d898-f089-47b7-938d-7f6c2d0339cf-konnectivity-ca\") pod \"konnectivity-agent-fz4s9\" (UID: \"c517d898-f089-47b7-938d-7f6c2d0339cf\") " pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-host-slash\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021823 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-modprobe-d\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-run-netns\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021888 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021914 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-cni-bin\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5hb\" (UniqueName: \"kubernetes.io/projected/708eea1b-63a0-46a1-b87d-42f8c7b3f318-kube-api-access-dw5hb\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.021988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-cni-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.022957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-cni-binary-copy\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022052 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cni-binary-copy\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022078 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-kubernetes\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022104 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-daemon-config\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022158 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022203 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022227 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-cni-netd\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022252 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-multus-certs\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-etc-kubernetes\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/451a8c3d-179c-4a38-a3c3-50c96043ebf1-host\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/451a8c3d-179c-4a38-a3c3-50c96043ebf1-serviceca\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022349 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-systemd\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-sys\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022398 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-k8s-cni-cncf-io\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-kubelet\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8a42fee-4675-42df-8f1a-71cb407b62ad-kubelet-config\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:45.023865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022474 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8a42fee-4675-42df-8f1a-71cb407b62ad-dbus\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022525 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c517d898-f089-47b7-938d-7f6c2d0339cf-agent-certs\") pod \"konnectivity-agent-fz4s9\" (UID: \"c517d898-f089-47b7-938d-7f6c2d0339cf\") " pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022551 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-lib-modules\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022577 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-var-lib-kubelet\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022625 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-conf-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-cni-binary-copy\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-daemon-config\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghzk\" (UniqueName: \"kubernetes.io/projected/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-kube-api-access-2ghzk\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-kubernetes\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.022766 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022821 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-var-lib-kubelet\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022829 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-etc-kubernetes\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022868 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/451a8c3d-179c-4a38-a3c3-50c96043ebf1-host\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8a42fee-4675-42df-8f1a-71cb407b62ad-kubelet-config\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.022896 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-cni-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023071 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-k8s-cni-cncf-io\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023097 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8a42fee-4675-42df-8f1a-71cb407b62ad-dbus\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:45.024577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023124 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-var-lib-kubelet\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023170 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-host-run-multus-certs\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023195 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-lib-modules\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-multus-conf-dir\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023257 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-systemd\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.023277 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret podName:e8a42fee-4675-42df-8f1a-71cb407b62ad nodeName:}" failed. No retries permitted until 2026-07-15 08:00:45.523252795 +0000 UTC m=+3.039049627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret") pod "global-pull-secret-syncer-nlzkz" (UID: "e8a42fee-4675-42df-8f1a-71cb407b62ad") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/708eea1b-63a0-46a1-b87d-42f8c7b3f318-sys\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.023340 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/451a8c3d-179c-4a38-a3c3-50c96043ebf1-serviceca\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.024272 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/708eea1b-63a0-46a1-b87d-42f8c7b3f318-etc-tuned\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.025383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.024786 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/708eea1b-63a0-46a1-b87d-42f8c7b3f318-tmp\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.028929 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.028647 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:45.028929 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.028671 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:45.028929 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.028688 2567 projected.go:196] Error preparing data for projected volume kube-api-access-99kdf for pod openshift-network-diagnostics/network-check-target-7tfd8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:45.028929 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.028750 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf podName:1318d214-3429-46bc-9a2b-bcbcab7e56d5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:45.528733387 +0000 UTC m=+3.044530218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-99kdf" (UniqueName: "kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf") pod "network-check-target-7tfd8" (UID: "1318d214-3429-46bc-9a2b-bcbcab7e56d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:45.034384 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.031823 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d28ck\" (UniqueName: \"kubernetes.io/projected/451a8c3d-179c-4a38-a3c3-50c96043ebf1-kube-api-access-d28ck\") pod \"node-ca-clz7b\" (UID: \"451a8c3d-179c-4a38-a3c3-50c96043ebf1\") " pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:45.034384 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.031921 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5hb\" (UniqueName: \"kubernetes.io/projected/708eea1b-63a0-46a1-b87d-42f8c7b3f318-kube-api-access-dw5hb\") pod \"tuned-bvrzh\" (UID: \"708eea1b-63a0-46a1-b87d-42f8c7b3f318\") " pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.034384 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.031954 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g946\" (UniqueName: \"kubernetes.io/projected/9623c37b-1793-4c8c-aaa8-aaaef9b0bded-kube-api-access-9g946\") pod \"multus-hk8lc\" (UID: \"9623c37b-1793-4c8c-aaa8-aaaef9b0bded\") " pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.034829 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.034769 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" event={"ID":"bf158e8b3088cd152953b639a569c880","Type":"ContainerStarted","Data":"8286f827de82ab66e2f92328971f2c2d2cb96a22973cd02066c2e520ecfb83d9"} Jul 15 08:00:45.060627 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.060606 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9545b0e517d36e2a7fb3813f2cac8af0.slice/crio-23fe5358c2d3c8afa161ac7d188c4365eb85e08d160deaaab21a3549ccf28731 WatchSource:0}: Error finding container 23fe5358c2d3c8afa161ac7d188c4365eb85e08d160deaaab21a3549ccf28731: Status 404 returned error can't find the container with id 23fe5358c2d3c8afa161ac7d188c4365eb85e08d160deaaab21a3549ccf28731 Jul 15 08:00:45.110802 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.110776 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:45.123124 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.123124 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c517d898-f089-47b7-938d-7f6c2d0339cf-konnectivity-ca\") pod \"konnectivity-agent-fz4s9\" (UID: \"c517d898-f089-47b7-938d-7f6c2d0339cf\") " pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:00:45.123124 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123111 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-host-slash\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.123124 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123127 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-run-netns\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123148 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123171 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-cni-bin\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123200 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-host-slash\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123235 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-cni-bin\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123241 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-run-netns\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cni-binary-copy\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123340 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123367 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-cni-netd\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c517d898-f089-47b7-938d-7f6c2d0339cf-agent-certs\") pod \"konnectivity-agent-fz4s9\" (UID: \"c517d898-f089-47b7-938d-7f6c2d0339cf\") " pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123462 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123524 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-cni-netd\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghzk\" (UniqueName: \"kubernetes.io/projected/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-kube-api-access-2ghzk\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123577 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-device-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-node-log\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-ovnkube-script-lib\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-hosts-file\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-registration-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123737 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-etc-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123727 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c517d898-f089-47b7-938d-7f6c2d0339cf-konnectivity-ca\") pod \"konnectivity-agent-fz4s9\" (UID: \"c517d898-f089-47b7-938d-7f6c2d0339cf\") " pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-ovn\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.123795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123794 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswf2\" (UniqueName: \"kubernetes.io/projected/bc482a89-5896-486c-ac28-9569d5b79c44-kube-api-access-rswf2\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123816 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123825 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-os-release\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123850 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-iptables-alerter-script\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123871 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-etc-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123875 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-log-socket\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123903 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123921 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-hosts-file\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-system-cni-dir\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123957 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-tmp-dir\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-registration-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.123983 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-var-lib-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124008 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cnibin\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124061 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-systemd-units\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-system-cni-dir\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc482a89-5896-486c-ac28-9569d5b79c44-ovn-node-metrics-cert\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124128 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9z9k\" (UniqueName: \"kubernetes.io/projected/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-kube-api-access-c9z9k\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.124453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-sys-fs\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8qj\" (UniqueName: \"kubernetes.io/projected/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-kube-api-access-7b8qj\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-kubelet\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124246 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4t84\" (UniqueName: \"kubernetes.io/projected/98640408-aff8-4f6d-8928-15afdf0e3ae3-kube-api-access-d4t84\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124274 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-slash\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124334 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-ovn\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-env-overrides\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfsxj\" (UniqueName: \"kubernetes.io/projected/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-kube-api-access-kfsxj\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124429 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-os-release\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124436 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-socket-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124465 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-systemd\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124490 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-iptables-alerter-script\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124501 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-device-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124513 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-ovnkube-config\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124607 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-log-socket\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.125391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cnibin\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124642 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124668 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-systemd-units\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124881 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-kubelet\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124885 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-tmp-dir\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.124955 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-sys-fs\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125016 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-ovnkube-script-lib\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125057 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-socket-dir\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-host-slash\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.125119 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125141 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-node-log\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.125188 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs podName:88bb6b5a-617c-4e10-871a-0060bbb1cdf3 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:45.625170092 +0000 UTC m=+3.140966910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs") pod "network-metrics-daemon-f6cf9" (UID: "88bb6b5a-617c-4e10-871a-0060bbb1cdf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-var-lib-openvswitch\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98640408-aff8-4f6d-8928-15afdf0e3ae3-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc482a89-5896-486c-ac28-9569d5b79c44-run-systemd\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125448 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-env-overrides\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126805 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125632 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-cni-binary-copy\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.126805 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.125734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc482a89-5896-486c-ac28-9569d5b79c44-ovnkube-config\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.126805 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.126595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c517d898-f089-47b7-938d-7f6c2d0339cf-agent-certs\") pod \"konnectivity-agent-fz4s9\" (UID: \"c517d898-f089-47b7-938d-7f6c2d0339cf\") " pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:00:45.127116 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.127097 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc482a89-5896-486c-ac28-9569d5b79c44-ovn-node-metrics-cert\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.134739 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.134711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghzk\" (UniqueName: \"kubernetes.io/projected/88e8e60f-c63e-43f8-9ac7-7cd0b3682693-kube-api-access-2ghzk\") pod \"multus-additional-cni-plugins-47mlt\" (UID: \"88e8e60f-c63e-43f8-9ac7-7cd0b3682693\") " pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.135366 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.135341 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8qj\" (UniqueName: \"kubernetes.io/projected/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-kube-api-access-7b8qj\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:45.135709 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.135660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4t84\" (UniqueName: \"kubernetes.io/projected/98640408-aff8-4f6d-8928-15afdf0e3ae3-kube-api-access-d4t84\") pod \"aws-ebs-csi-driver-node-8lz4k\" (UID: \"98640408-aff8-4f6d-8928-15afdf0e3ae3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.135807 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.135718 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfsxj\" (UniqueName: \"kubernetes.io/projected/7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2-kube-api-access-kfsxj\") pod \"node-resolver-txzrm\" (UID: \"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2\") " pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.136805 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.136779 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9z9k\" (UniqueName: \"kubernetes.io/projected/4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6-kube-api-access-c9z9k\") pod \"iptables-alerter-57h8c\" (UID: \"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6\") " pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.137051 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.137033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswf2\" (UniqueName: \"kubernetes.io/projected/bc482a89-5896-486c-ac28-9569d5b79c44-kube-api-access-rswf2\") pod \"ovnkube-node-bnbv5\" (UID: \"bc482a89-5896-486c-ac28-9569d5b79c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.187827 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.187794 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:45.192179 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.192158 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" Jul 15 08:00:45.199879 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.199860 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk8lc" Jul 15 08:00:45.203416 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.203380 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708eea1b_63a0_46a1_b87d_42f8c7b3f318.slice/crio-2cf2a5c940c81488445ca7540ba42f763b60518890667168b62124d2ed97964d WatchSource:0}: Error finding container 2cf2a5c940c81488445ca7540ba42f763b60518890667168b62124d2ed97964d: Status 404 returned error can't find the container with id 2cf2a5c940c81488445ca7540ba42f763b60518890667168b62124d2ed97964d Jul 15 08:00:45.208231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.208209 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-clz7b" Jul 15 08:00:45.208718 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.208692 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9623c37b_1793_4c8c_aaa8_aaaef9b0bded.slice/crio-63cee8fd8b1f1871036e8efe1621165926395072df4b5776fcc5ae32e8211ebd WatchSource:0}: Error finding container 63cee8fd8b1f1871036e8efe1621165926395072df4b5776fcc5ae32e8211ebd: Status 404 returned error can't find the container with id 63cee8fd8b1f1871036e8efe1621165926395072df4b5776fcc5ae32e8211ebd Jul 15 08:00:45.214476 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.214437 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:00:45.218826 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.218794 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451a8c3d_179c_4a38_a3c3_50c96043ebf1.slice/crio-d30037b55f3a3aef2d46feccd5864345b03ecdc47c34757cf38b453e0a7db248 WatchSource:0}: Error finding container d30037b55f3a3aef2d46feccd5864345b03ecdc47c34757cf38b453e0a7db248: Status 404 returned error can't find the container with id d30037b55f3a3aef2d46feccd5864345b03ecdc47c34757cf38b453e0a7db248 Jul 15 08:00:45.222197 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.222175 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-txzrm" Jul 15 08:00:45.225894 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.225870 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc482a89_5896_486c_ac28_9569d5b79c44.slice/crio-77735e470516704446a71810c7d0eb4439291331bc6084a32c032709f75a4d68 WatchSource:0}: Error finding container 77735e470516704446a71810c7d0eb4439291331bc6084a32c032709f75a4d68: Status 404 returned error can't find the container with id 77735e470516704446a71810c7d0eb4439291331bc6084a32c032709f75a4d68 Jul 15 08:00:45.230072 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.230049 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-47mlt" Jul 15 08:00:45.231226 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.231203 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5df8c5_cdfe_4f96_b7a9_50f5a4bad2c2.slice/crio-54e854d115316f2adc6f953ed2c3c3be8b9e56e38ae724f90e4824f02c2fe002 WatchSource:0}: Error finding container 54e854d115316f2adc6f953ed2c3c3be8b9e56e38ae724f90e4824f02c2fe002: Status 404 returned error can't find the container with id 54e854d115316f2adc6f953ed2c3c3be8b9e56e38ae724f90e4824f02c2fe002 Jul 15 08:00:45.239580 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.239562 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:00:45.239877 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.239849 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e8e60f_c63e_43f8_9ac7_7cd0b3682693.slice/crio-0a5d00e975b82e68d83296c5a2212ffb67ba5fa96dcd22e6b0260dc7b405a957 WatchSource:0}: Error finding container 0a5d00e975b82e68d83296c5a2212ffb67ba5fa96dcd22e6b0260dc7b405a957: Status 404 returned error can't find the container with id 0a5d00e975b82e68d83296c5a2212ffb67ba5fa96dcd22e6b0260dc7b405a957 Jul 15 08:00:45.247081 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.247060 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57h8c" Jul 15 08:00:45.247302 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.247285 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc517d898_f089_47b7_938d_7f6c2d0339cf.slice/crio-b65f2d3eafc4402124bbcafd608e8ba936bd0454ec839d0e6bcdefc1584f19ae WatchSource:0}: Error finding container b65f2d3eafc4402124bbcafd608e8ba936bd0454ec839d0e6bcdefc1584f19ae: Status 404 returned error can't find the container with id b65f2d3eafc4402124bbcafd608e8ba936bd0454ec839d0e6bcdefc1584f19ae Jul 15 08:00:45.254705 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.254686 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" Jul 15 08:00:45.527691 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.527600 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:45.527842 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.527726 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:45.527842 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.527793 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret podName:e8a42fee-4675-42df-8f1a-71cb407b62ad nodeName:}" failed. No retries permitted until 2026-07-15 08:00:46.527775609 +0000 UTC m=+4.043572446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret") pod "global-pull-secret-syncer-nlzkz" (UID: "e8a42fee-4675-42df-8f1a-71cb407b62ad") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:45.590743 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.590713 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98640408_aff8_4f6d_8928_15afdf0e3ae3.slice/crio-045b989c1659187671533b074c17d89a43ff389e0b77ed761747da5302b2c808 WatchSource:0}: Error finding container 045b989c1659187671533b074c17d89a43ff389e0b77ed761747da5302b2c808: Status 404 returned error can't find the container with id 045b989c1659187671533b074c17d89a43ff389e0b77ed761747da5302b2c808 Jul 15 08:00:45.592180 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:00:45.592157 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c505e6e_e2eb_41b4_9fcf_453ba1d9faa6.slice/crio-2e36467e22d9e9a63db1667e0d423b18c97e07a1e59c310bf74b5ee6c807090f WatchSource:0}: Error finding container 2e36467e22d9e9a63db1667e0d423b18c97e07a1e59c310bf74b5ee6c807090f: Status 404 returned error can't find the container with id 2e36467e22d9e9a63db1667e0d423b18c97e07a1e59c310bf74b5ee6c807090f Jul 15 08:00:45.628543 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.628501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:45.628652 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.628549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:45.628652 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.628647 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:45.628842 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.628659 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:45.628842 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.628675 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:45.628842 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.628688 2567 projected.go:196] Error preparing data for projected volume kube-api-access-99kdf for pod openshift-network-diagnostics/network-check-target-7tfd8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:45.628842 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.628707 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs podName:88bb6b5a-617c-4e10-871a-0060bbb1cdf3 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:46.628686804 +0000 UTC m=+4.144483636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs") pod "network-metrics-daemon-f6cf9" (UID: "88bb6b5a-617c-4e10-871a-0060bbb1cdf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:45.628842 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:45.628728 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf podName:1318d214-3429-46bc-9a2b-bcbcab7e56d5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:46.628713651 +0000 UTC m=+4.144510468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-99kdf" (UniqueName: "kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf") pod "network-check-target-7tfd8" (UID: "1318d214-3429-46bc-9a2b-bcbcab7e56d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:45.933613 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.933341 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 07:55:45 +0000 UTC" deadline="2028-04-14 08:34:05.39319106 +0000 UTC" Jul 15 08:00:45.933613 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:45.933565 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15336h33m19.45963183s" Jul 15 08:00:46.028687 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.028654 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:46.028865 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.028816 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:00:46.042994 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.042257 2567 generic.go:356] "Generic (PLEG): container finished" podID="bf158e8b3088cd152953b639a569c880" containerID="ebdd5be2e9dbb98ef1866bb1c16fdde8e907967735c38ac65efd8a8ad0ec068c" exitCode=0 Jul 15 08:00:46.042994 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.042385 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" event={"ID":"bf158e8b3088cd152953b639a569c880","Type":"ContainerDied","Data":"ebdd5be2e9dbb98ef1866bb1c16fdde8e907967735c38ac65efd8a8ad0ec068c"} Jul 15 08:00:46.046650 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.046626 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57h8c" event={"ID":"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6","Type":"ContainerStarted","Data":"2e36467e22d9e9a63db1667e0d423b18c97e07a1e59c310bf74b5ee6c807090f"} Jul 15 08:00:46.050665 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.050559 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fz4s9" event={"ID":"c517d898-f089-47b7-938d-7f6c2d0339cf","Type":"ContainerStarted","Data":"b65f2d3eafc4402124bbcafd608e8ba936bd0454ec839d0e6bcdefc1584f19ae"} Jul 15 08:00:46.058789 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.058758 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-clz7b" event={"ID":"451a8c3d-179c-4a38-a3c3-50c96043ebf1","Type":"ContainerStarted","Data":"d30037b55f3a3aef2d46feccd5864345b03ecdc47c34757cf38b453e0a7db248"} Jul 15 08:00:46.062792 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.062669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" event={"ID":"708eea1b-63a0-46a1-b87d-42f8c7b3f318","Type":"ContainerStarted","Data":"2cf2a5c940c81488445ca7540ba42f763b60518890667168b62124d2ed97964d"} Jul 15 08:00:46.066365 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.066156 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" event={"ID":"9545b0e517d36e2a7fb3813f2cac8af0","Type":"ContainerStarted","Data":"23fe5358c2d3c8afa161ac7d188c4365eb85e08d160deaaab21a3549ccf28731"} Jul 15 08:00:46.073056 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.072994 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" event={"ID":"98640408-aff8-4f6d-8928-15afdf0e3ae3","Type":"ContainerStarted","Data":"045b989c1659187671533b074c17d89a43ff389e0b77ed761747da5302b2c808"} Jul 15 08:00:46.079168 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.079146 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47mlt" event={"ID":"88e8e60f-c63e-43f8-9ac7-7cd0b3682693","Type":"ContainerStarted","Data":"0a5d00e975b82e68d83296c5a2212ffb67ba5fa96dcd22e6b0260dc7b405a957"} Jul 15 08:00:46.096698 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.095386 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-txzrm" event={"ID":"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2","Type":"ContainerStarted","Data":"54e854d115316f2adc6f953ed2c3c3be8b9e56e38ae724f90e4824f02c2fe002"} Jul 15 08:00:46.102496 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.102435 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"77735e470516704446a71810c7d0eb4439291331bc6084a32c032709f75a4d68"} Jul 15 08:00:46.110986 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.110942 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk8lc" event={"ID":"9623c37b-1793-4c8c-aaa8-aaaef9b0bded","Type":"ContainerStarted","Data":"63cee8fd8b1f1871036e8efe1621165926395072df4b5776fcc5ae32e8211ebd"} Jul 15 08:00:46.538642 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.538542 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:46.538809 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.538790 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:46.538885 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.538865 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret podName:e8a42fee-4675-42df-8f1a-71cb407b62ad nodeName:}" failed. No retries permitted until 2026-07-15 08:00:48.538843035 +0000 UTC m=+6.054639851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret") pod "global-pull-secret-syncer-nlzkz" (UID: "e8a42fee-4675-42df-8f1a-71cb407b62ad") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:46.639836 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.639795 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:46.640032 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.639852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:46.640099 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.640069 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:46.640099 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.640089 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:46.640209 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.640103 2567 projected.go:196] Error preparing data for projected volume kube-api-access-99kdf for pod openshift-network-diagnostics/network-check-target-7tfd8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:46.640209 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.640167 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf podName:1318d214-3429-46bc-9a2b-bcbcab7e56d5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:48.640147154 +0000 UTC m=+6.155943971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-99kdf" (UniqueName: "kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf") pod "network-check-target-7tfd8" (UID: "1318d214-3429-46bc-9a2b-bcbcab7e56d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:46.640698 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.640596 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:46.640698 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:46.640656 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs podName:88bb6b5a-617c-4e10-871a-0060bbb1cdf3 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:48.640638163 +0000 UTC m=+6.156434980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs") pod "network-metrics-daemon-f6cf9" (UID: "88bb6b5a-617c-4e10-871a-0060bbb1cdf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:46.934782 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.934690 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 07:55:45 +0000 UTC" deadline="2028-02-18 10:58:11.510582997 +0000 UTC" Jul 15 08:00:46.934782 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:46.934750 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13994h57m24.575836909s" Jul 15 08:00:47.028996 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:47.028242 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:47.028996 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:47.028390 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:00:47.028996 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:47.028824 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:47.028996 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:47.028937 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:00:47.141548 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:47.141016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" event={"ID":"bf158e8b3088cd152953b639a569c880","Type":"ContainerStarted","Data":"5e7ef9a86ba3af6392db0a30e4d868c27b4db6610ff192d13d7f9eb13fb89337"} Jul 15 08:00:48.028270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:48.028238 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:48.028735 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.028369 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:00:48.555259 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:48.555222 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:48.555463 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.555374 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:48.555539 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.555501 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret podName:e8a42fee-4675-42df-8f1a-71cb407b62ad nodeName:}" failed. No retries permitted until 2026-07-15 08:00:52.555481571 +0000 UTC m=+10.071278386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret") pod "global-pull-secret-syncer-nlzkz" (UID: "e8a42fee-4675-42df-8f1a-71cb407b62ad") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:48.656606 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:48.655832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:48.656606 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:48.655883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:48.656606 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.656037 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:48.656606 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.656058 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:48.656606 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.656071 2567 projected.go:196] Error preparing data for projected volume kube-api-access-99kdf for pod openshift-network-diagnostics/network-check-target-7tfd8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:48.656606 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.656127 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf podName:1318d214-3429-46bc-9a2b-bcbcab7e56d5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:52.656109158 +0000 UTC m=+10.171905974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-99kdf" (UniqueName: "kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf") pod "network-check-target-7tfd8" (UID: "1318d214-3429-46bc-9a2b-bcbcab7e56d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:48.656606 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.656521 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:48.656606 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:48.656574 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs podName:88bb6b5a-617c-4e10-871a-0060bbb1cdf3 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:52.656556889 +0000 UTC m=+10.172353704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs") pod "network-metrics-daemon-f6cf9" (UID: "88bb6b5a-617c-4e10-871a-0060bbb1cdf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:49.032891 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:49.032864 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:49.033340 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:49.032975 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:00:49.033430 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:49.033345 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:49.033548 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:49.033473 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:00:50.027904 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:50.027875 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:50.028050 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:50.027995 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:00:51.028551 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:51.028472 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:51.028966 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:51.028621 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:00:51.029039 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:51.029021 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:51.029158 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:51.029136 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:00:52.028326 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:52.028289 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:52.028508 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.028428 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:00:52.592639 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:52.592602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:52.593116 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.592759 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:52.593116 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.592839 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret podName:e8a42fee-4675-42df-8f1a-71cb407b62ad nodeName:}" failed. No retries permitted until 2026-07-15 08:01:00.592818765 +0000 UTC m=+18.108615581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret") pod "global-pull-secret-syncer-nlzkz" (UID: "e8a42fee-4675-42df-8f1a-71cb407b62ad") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:52.693624 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:52.693281 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:52.693624 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:52.693328 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:52.693624 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.693457 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:52.693624 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.693509 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:52.693624 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.693526 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:52.693624 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.693530 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs podName:88bb6b5a-617c-4e10-871a-0060bbb1cdf3 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:00.693507399 +0000 UTC m=+18.209304235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs") pod "network-metrics-daemon-f6cf9" (UID: "88bb6b5a-617c-4e10-871a-0060bbb1cdf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:52.693624 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.693546 2567 projected.go:196] Error preparing data for projected volume kube-api-access-99kdf for pod openshift-network-diagnostics/network-check-target-7tfd8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:52.693624 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:52.693595 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf podName:1318d214-3429-46bc-9a2b-bcbcab7e56d5 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:00.693580137 +0000 UTC m=+18.209376963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-99kdf" (UniqueName: "kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf") pod "network-check-target-7tfd8" (UID: "1318d214-3429-46bc-9a2b-bcbcab7e56d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:53.030174 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:53.029561 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:53.030174 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:53.029669 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:00:53.030174 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:53.030023 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:53.030174 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:53.030134 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:00:54.028116 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:54.028082 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:54.028579 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:54.028211 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:00:55.028746 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:55.028713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:55.029193 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:55.028840 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:00:55.029193 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:55.028891 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:55.029193 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:55.029019 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:00:56.028307 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:56.028276 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:56.028474 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:56.028398 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:00:57.028783 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:57.028705 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:57.029184 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:57.028815 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:00:57.029184 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:57.028846 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:57.029184 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:57.028929 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:00:58.028662 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:58.028613 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:00:58.028825 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:58.028735 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:00:59.028759 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:59.028724 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:00:59.028928 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:59.028854 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:00:59.028928 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:00:59.028894 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:00:59.029319 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:00:59.028959 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:00.028629 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:00.028591 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:00.028822 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.028706 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:00.652012 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:00.651976 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:00.652369 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.652112 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:01:00.652369 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.652190 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret podName:e8a42fee-4675-42df-8f1a-71cb407b62ad nodeName:}" failed. No retries permitted until 2026-07-15 08:01:16.652168525 +0000 UTC m=+34.167965342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret") pod "global-pull-secret-syncer-nlzkz" (UID: "e8a42fee-4675-42df-8f1a-71cb407b62ad") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:01:00.752811 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:00.752776 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:00.752966 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:00.752822 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:00.752966 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.752946 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:01:00.752966 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.752961 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:01:00.753076 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.752978 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:01:00.753076 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.752990 2567 projected.go:196] Error preparing data for projected volume kube-api-access-99kdf for pod openshift-network-diagnostics/network-check-target-7tfd8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:01:00.753076 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.753016 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs podName:88bb6b5a-617c-4e10-871a-0060bbb1cdf3 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:16.752994349 +0000 UTC m=+34.268791175 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs") pod "network-metrics-daemon-f6cf9" (UID: "88bb6b5a-617c-4e10-871a-0060bbb1cdf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:01:00.753076 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:00.753038 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf podName:1318d214-3429-46bc-9a2b-bcbcab7e56d5 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:16.753023028 +0000 UTC m=+34.268819855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-99kdf" (UniqueName: "kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf") pod "network-check-target-7tfd8" (UID: "1318d214-3429-46bc-9a2b-bcbcab7e56d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:01:01.028769 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:01.028683 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:01.028769 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:01.028728 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:01.028953 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:01.028838 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:01.028953 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:01.028864 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:02.028031 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:02.027997 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:02.028478 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:02.028115 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:03.029048 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:03.029010 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:03.029503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:03.029148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:03.029503 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:03.029148 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:03.029503 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:03.029256 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:04.028145 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:04.028114 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:04.028288 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:04.028223 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:05.027873 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.027735 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:05.028431 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.027803 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:05.028431 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:05.027973 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:05.028431 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:05.028017 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:05.178883 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.178843 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" event={"ID":"9545b0e517d36e2a7fb3813f2cac8af0","Type":"ContainerStarted","Data":"2ce8f1202e704fcb9ec643fd376d338a5f6833ca2b27b84994a312cc67591afe"} Jul 15 08:01:05.180307 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.180285 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" event={"ID":"98640408-aff8-4f6d-8928-15afdf0e3ae3","Type":"ContainerStarted","Data":"3c1339bd164e322e7cc4ee59990b9e06ad7c351c5ed6a4dcd9196155eecec6ba"} Jul 15 08:01:05.181747 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.181722 2567 generic.go:356] "Generic (PLEG): container finished" podID="88e8e60f-c63e-43f8-9ac7-7cd0b3682693" containerID="338546776a5eadbf0648cc0b108f479dc4e9c3a53b033610253383f62a894b14" exitCode=0 Jul 15 08:01:05.181834 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.181801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47mlt" event={"ID":"88e8e60f-c63e-43f8-9ac7-7cd0b3682693","Type":"ContainerDied","Data":"338546776a5eadbf0648cc0b108f479dc4e9c3a53b033610253383f62a894b14"} Jul 15 08:01:05.183416 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.183395 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-txzrm" event={"ID":"7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2","Type":"ContainerStarted","Data":"93525a823fa9c8673783e81e818c70657b9279691e23aa5e097502c32bc44073"} Jul 15 08:01:05.185305 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.185290 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:01:05.185754 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.185730 2567 generic.go:356] "Generic (PLEG): container finished" podID="bc482a89-5896-486c-ac28-9569d5b79c44" containerID="27f9b8e135f399197bb0a1a379c62cc93b7842d84022cc8adba9db27f64d83c1" exitCode=1 Jul 15 08:01:05.185814 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.185793 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"48cde509b5798b4be407fae757602e68adaae9250edfce5fe5ec516c9a0c0156"} Jul 15 08:01:05.185874 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.185861 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerDied","Data":"27f9b8e135f399197bb0a1a379c62cc93b7842d84022cc8adba9db27f64d83c1"} Jul 15 08:01:05.185928 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.185903 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"3e9749f852386fe46c6c851dd0ccf83616461d15b7e327d48ec0f5241d865ef4"} Jul 15 08:01:05.187220 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.187198 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk8lc" event={"ID":"9623c37b-1793-4c8c-aaa8-aaaef9b0bded","Type":"ContainerStarted","Data":"0fa12eb8d5962aa98bbaa941d67975846d397c6412f0e8cf5e03a7fde95d7d56"} Jul 15 08:01:05.188677 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.188658 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fz4s9" event={"ID":"c517d898-f089-47b7-938d-7f6c2d0339cf","Type":"ContainerStarted","Data":"ca64b071046a9961663823cb398b2b55ce933fa65d59d05bf70afb4297aab804"} Jul 15 08:01:05.190083 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.190062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-clz7b" event={"ID":"451a8c3d-179c-4a38-a3c3-50c96043ebf1","Type":"ContainerStarted","Data":"68b13e3010938392b62e2f26ca827031ac2d41ebaf4c7ee2f24b60dee44b848f"} Jul 15 08:01:05.191508 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.191489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" event={"ID":"708eea1b-63a0-46a1-b87d-42f8c7b3f318","Type":"ContainerStarted","Data":"2c3080c041fb02411c37c67c7d8ce55689263c96c367e1f8af8c4c1337311cd7"} Jul 15 08:01:05.194794 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.194743 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" podStartSLOduration=21.19472491 podStartE2EDuration="21.19472491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:05.194143611 +0000 UTC m=+22.709940445" watchObservedRunningTime="2026-07-15 08:01:05.19472491 +0000 UTC m=+22.710521746" Jul 15 08:01:05.195061 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.195033 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" podStartSLOduration=21.195025782 podStartE2EDuration="21.195025782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:47.163910368 +0000 UTC m=+4.679707206" watchObservedRunningTime="2026-07-15 08:01:05.195025782 +0000 UTC m=+22.710822616" Jul 15 08:01:05.235982 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.235944 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bvrzh" podStartSLOduration=2.726316423 podStartE2EDuration="22.235927469s" totalImagesPullingTime="19.509611046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:05.23583102 +0000 UTC m=+22.751627858" watchObservedRunningTime="2026-07-15 08:01:05.235927469 +0000 UTC m=+22.751724309" Jul 15 08:01:05.250211 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.250169 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-clz7b" podStartSLOduration=2.7608789099999997 podStartE2EDuration="22.250152933s" totalImagesPullingTime="19.489274023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:05.249830827 +0000 UTC m=+22.765627664" watchObservedRunningTime="2026-07-15 08:01:05.250152933 +0000 UTC m=+22.765949766" Jul 15 08:01:05.288327 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.288288 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-txzrm" podStartSLOduration=2.809858222 podStartE2EDuration="22.28826738s" totalImagesPullingTime="19.478409158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:05.268165991 +0000 UTC m=+22.783962824" watchObservedRunningTime="2026-07-15 08:01:05.28826738 +0000 UTC m=+22.804064214" Jul 15 08:01:05.307412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.307374 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fz4s9" podStartSLOduration=2.847324502 podStartE2EDuration="22.307359832s" totalImagesPullingTime="19.46003533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:05.288159652 +0000 UTC m=+22.803956487" watchObservedRunningTime="2026-07-15 08:01:05.307359832 +0000 UTC m=+22.823156669" Jul 15 08:01:05.307699 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.307679 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hk8lc" podStartSLOduration=2.756960028 podStartE2EDuration="22.307674191s" totalImagesPullingTime="19.550714163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:05.307096299 +0000 UTC m=+22.822893135" watchObservedRunningTime="2026-07-15 08:01:05.307674191 +0000 UTC m=+22.823471024" Jul 15 08:01:05.411181 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.411152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:05.673303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:05.673105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:06.010911 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.010873 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:06.027882 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.027861 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:06.028204 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:06.027971 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:06.196051 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.196031 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:01:06.196378 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.196355 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"607acca655e2ea1aaab73d51b4158cc627df41bfaab93eb30d640d13ab94f8cf"} Jul 15 08:01:06.196480 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.196454 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"cc503001e61e2e661c8547c3125996c8aa71acdd481d655ab4dcbebf6962e3de"} Jul 15 08:01:06.196546 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.196507 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"f8278a263edbc9d12bb4014e0e80794ba51cde3babd83c5b02048d45b304f8cf"} Jul 15 08:01:06.197676 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.197654 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57h8c" event={"ID":"4c505e6e-e2eb-41b4-9fcf-453ba1d9faa6","Type":"ContainerStarted","Data":"b18013167512e31e996aa4d6058d5553a06a8fc2ae4f194f1c3205f34e30c286"} Jul 15 08:01:06.215263 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.215200 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-57h8c" podStartSLOduration=4.09961201 podStartE2EDuration="23.215188169s" totalImagesPullingTime="19.115576159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:06.214997532 +0000 UTC m=+23.730794367" watchObservedRunningTime="2026-07-15 08:01:06.215188169 +0000 UTC m=+23.730985002" Jul 15 08:01:06.337999 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.337966 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:06.437903 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:06.437878 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:07.027725 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:07.027694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:07.027912 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:07.027859 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:07.028274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:07.027914 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:07.028274 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:07.028031 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:07.661731 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:07.661685 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 08:01:07.966296 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:07.966203 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T08:01:07.661705686Z","UUID":"1a1b6f88-5ca2-47b2-95cf-8d27f97ca561","Handler":null,"Name":"","Endpoint":""} Jul 15 08:01:07.968115 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:07.968086 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 08:01:07.968257 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:07.968131 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 08:01:08.027915 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:08.027890 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:08.028280 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:08.028002 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:08.201792 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:08.201764 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:01:08.202688 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:08.202663 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:01:08.205007 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:08.204965 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" event={"ID":"98640408-aff8-4f6d-8928-15afdf0e3ae3","Type":"ContainerStarted","Data":"3c85a5c7c91b48cb4f20e0bed3a8fcc3eabf21f1a826d0cb81023266fc36dc07"} Jul 15 08:01:08.209389 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:08.209368 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:01:08.209856 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:08.209828 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"16cc06dfdc4f2020211dc9ecb054d0c236b83312a34d87017591e1da53d7f65d"} Jul 15 08:01:09.028647 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:09.028453 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:09.029053 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:09.028495 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:09.029053 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:09.028745 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:09.029053 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:09.028821 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:09.211811 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:09.211781 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:01:09.212335 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:09.212312 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fz4s9" Jul 15 08:01:09.822648 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:09.822614 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:10.027926 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:10.027857 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:10.028062 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:10.027949 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:10.424898 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:10.424872 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:10.532123 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:10.532096 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:11.028582 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.028353 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:11.028732 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.028372 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:11.028732 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:11.028597 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:11.028732 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:11.028717 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:11.217192 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.217162 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" event={"ID":"98640408-aff8-4f6d-8928-15afdf0e3ae3","Type":"ContainerStarted","Data":"2330118aeeb86c1c7cb96b101d3913f14e67588bc1e6046896d27a76758fc787"} Jul 15 08:01:11.218771 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.218746 2567 generic.go:356] "Generic (PLEG): container finished" podID="88e8e60f-c63e-43f8-9ac7-7cd0b3682693" containerID="37b4ffa0d06d402cd29e01ef45a2971d3ee4c8bdc8d83224c50bbc03da311ddd" exitCode=0 Jul 15 08:01:11.218891 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.218827 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47mlt" event={"ID":"88e8e60f-c63e-43f8-9ac7-7cd0b3682693","Type":"ContainerDied","Data":"37b4ffa0d06d402cd29e01ef45a2971d3ee4c8bdc8d83224c50bbc03da311ddd"} Jul 15 08:01:11.222280 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.222265 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:01:11.222687 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.222669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"c15e65be28365bbb852bf033d4cb67deb845759b54be33d1852db5fd42191814"} Jul 15 08:01:11.222973 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.222955 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:01:11.223063 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.223025 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:01:11.223063 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.223043 2567 scope.go:117] "RemoveContainer" containerID="27f9b8e135f399197bb0a1a379c62cc93b7842d84022cc8adba9db27f64d83c1" Jul 15 08:01:11.223155 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.223073 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:01:11.239475 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.239391 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lz4k" podStartSLOduration=5.342009098 podStartE2EDuration="28.239372612s" totalImagesPullingTime="22.897363514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:11.23921343 +0000 UTC m=+28.755010264" watchObservedRunningTime="2026-07-15 08:01:11.239372612 +0000 UTC m=+28.755169448" Jul 15 08:01:11.239719 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.239702 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:01:11.239881 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:11.239863 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:01:12.028456 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.028408 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:12.029023 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:12.028562 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:12.228458 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.228422 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:01:12.228804 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.228780 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" event={"ID":"bc482a89-5896-486c-ac28-9569d5b79c44","Type":"ContainerStarted","Data":"24830ce8c3aad880d695cb38165e610109c13e2eb70f1439ab324e215f220b4d"} Jul 15 08:01:12.649897 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.649819 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" podStartSLOduration=10.083057691 podStartE2EDuration="29.649801494s" totalImagesPullingTime="19.566743803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:12.279269139 +0000 UTC m=+29.795065995" watchObservedRunningTime="2026-07-15 08:01:12.649801494 +0000 UTC m=+30.165598332" Jul 15 08:01:12.650020 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.649996 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f6cf9"] Jul 15 08:01:12.650100 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.650090 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:12.650194 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:12.650179 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:12.655901 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.655861 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nlzkz"] Jul 15 08:01:12.656017 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.655993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:12.656125 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:12.656097 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:12.656668 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.656642 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7tfd8"] Jul 15 08:01:12.656799 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.656730 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:12.656867 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:12.656816 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:12.734086 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:12.734064 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:13.233060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:13.233035 2567 generic.go:356] "Generic (PLEG): container finished" podID="88e8e60f-c63e-43f8-9ac7-7cd0b3682693" containerID="b690bc143987ebb126216e5b0bd436c1d798a7ff1b291ce8480efc6ebce809a7" exitCode=0 Jul 15 08:01:13.233388 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:13.233108 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47mlt" event={"ID":"88e8e60f-c63e-43f8-9ac7-7cd0b3682693","Type":"ContainerDied","Data":"b690bc143987ebb126216e5b0bd436c1d798a7ff1b291ce8480efc6ebce809a7"} Jul 15 08:01:13.342180 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:13.342147 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:14.028920 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:14.028670 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:14.029045 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:14.028718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:14.029045 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:14.029001 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:14.029117 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:14.029040 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:15.027750 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:15.027695 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:15.028144 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:15.027776 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:15.239611 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:15.239583 2567 generic.go:356] "Generic (PLEG): container finished" podID="88e8e60f-c63e-43f8-9ac7-7cd0b3682693" containerID="0d10a26083a47b5c469f720d72faf2b4720b7cbada8e4923b8977afc91592894" exitCode=0 Jul 15 08:01:15.239732 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:15.239625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47mlt" event={"ID":"88e8e60f-c63e-43f8-9ac7-7cd0b3682693","Type":"ContainerDied","Data":"0d10a26083a47b5c469f720d72faf2b4720b7cbada8e4923b8977afc91592894"} Jul 15 08:01:15.637209 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:15.637176 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:15.699889 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:15.699865 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:16.028664 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.028590 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:16.029177 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.028716 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6cf9" podUID="88bb6b5a-617c-4e10-871a-0060bbb1cdf3" Jul 15 08:01:16.029177 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.028773 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:16.029177 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.028864 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlzkz" podUID="e8a42fee-4675-42df-8f1a-71cb407b62ad" Jul 15 08:01:16.407717 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.407679 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:16.486135 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.486103 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:16.673914 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.673845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:16.674059 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.673977 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:01:16.674059 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.674036 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret podName:e8a42fee-4675-42df-8f1a-71cb407b62ad nodeName:}" failed. No retries permitted until 2026-07-15 08:01:48.674018869 +0000 UTC m=+66.189815682 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret") pod "global-pull-secret-syncer-nlzkz" (UID: "e8a42fee-4675-42df-8f1a-71cb407b62ad") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:01:16.749273 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.749244 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:16.774274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.774239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:16.774412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.774289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:16.774412 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.774360 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:01:16.774571 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.774449 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs podName:88bb6b5a-617c-4e10-871a-0060bbb1cdf3 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:48.774415347 +0000 UTC m=+66.290212173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs") pod "network-metrics-daemon-f6cf9" (UID: "88bb6b5a-617c-4e10-871a-0060bbb1cdf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:01:16.774571 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.774480 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:01:16.774571 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.774499 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:01:16.774571 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.774511 2567 projected.go:196] Error preparing data for projected volume kube-api-access-99kdf for pod openshift-network-diagnostics/network-check-target-7tfd8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:01:16.774571 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:16.774568 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf podName:1318d214-3429-46bc-9a2b-bcbcab7e56d5 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:48.774551076 +0000 UTC m=+66.290347908 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-99kdf" (UniqueName: "kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf") pod "network-check-target-7tfd8" (UID: "1318d214-3429-46bc-9a2b-bcbcab7e56d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:01:16.822547 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:16.822519 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:17.028072 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.028005 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:17.028216 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:17.028111 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7tfd8" podUID="1318d214-3429-46bc-9a2b-bcbcab7e56d5" Jul 15 08:01:17.157369 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.157337 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:17.360372 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.360329 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:17.455423 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.455394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:17.557350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.557318 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:17.744389 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.744321 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeReady" Jul 15 08:01:17.744553 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.744483 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 08:01:17.799921 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.799891 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m8h8b"] Jul 15 08:01:17.801844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.801822 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.806550 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.806509 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 08:01:17.806811 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.806779 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 08:01:17.806970 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.806940 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8tgdm\"" Jul 15 08:01:17.807450 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.807419 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x7gwv"] Jul 15 08:01:17.808994 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.808974 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7gwv" Jul 15 08:01:17.811215 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.811190 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 08:01:17.811317 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.811296 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lkxcp\"" Jul 15 08:01:17.811474 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.811407 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 08:01:17.811548 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.811483 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 08:01:17.816492 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.816470 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m8h8b"] Jul 15 08:01:17.822326 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.822305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7gwv"] Jul 15 08:01:17.828117 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.828092 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lnlh8"] Jul 15 08:01:17.830098 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.830078 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.833044 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.833021 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f2tcw\"" Jul 15 08:01:17.833134 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.833050 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 08:01:17.833243 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.833221 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 08:01:17.833294 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.833262 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 08:01:17.833784 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.833767 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 08:01:17.844022 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.844006 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lnlh8"] Jul 15 08:01:17.882865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.882839 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18bef297-7e7a-40d4-ac55-291487d73ee7-data-volume\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.883029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.882871 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00bae080-60c7-49c2-aff2-93e41d0e23be-config-volume\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.883029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.882902 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.883029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.882928 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00bae080-60c7-49c2-aff2-93e41d0e23be-tmp-dir\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.883029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.882981 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.883029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.883022 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bvb\" (UniqueName: \"kubernetes.io/projected/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-api-access-98bvb\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.883274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.883045 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18bef297-7e7a-40d4-ac55-291487d73ee7-crio-socket\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.883274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.883092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b07d8157-8401-4516-ac0d-20c2d5c65275-cert\") pod \"ingress-canary-x7gwv\" (UID: \"b07d8157-8401-4516-ac0d-20c2d5c65275\") " pod="openshift-ingress-canary/ingress-canary-x7gwv" Jul 15 08:01:17.883274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.883124 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxpc\" (UniqueName: \"kubernetes.io/projected/00bae080-60c7-49c2-aff2-93e41d0e23be-kube-api-access-zlxpc\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.883274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.883152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.883274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.883176 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n762h\" (UniqueName: \"kubernetes.io/projected/b07d8157-8401-4516-ac0d-20c2d5c65275-kube-api-access-n762h\") pod \"ingress-canary-x7gwv\" (UID: \"b07d8157-8401-4516-ac0d-20c2d5c65275\") " pod="openshift-ingress-canary/ingress-canary-x7gwv" Jul 15 08:01:17.983464 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983399 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.983464 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n762h\" (UniqueName: \"kubernetes.io/projected/b07d8157-8401-4516-ac0d-20c2d5c65275-kube-api-access-n762h\") pod \"ingress-canary-x7gwv\" (UID: \"b07d8157-8401-4516-ac0d-20c2d5c65275\") " pod="openshift-ingress-canary/ingress-canary-x7gwv" Jul 15 08:01:17.983698 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:17.983531 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:17.983698 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:17.983588 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:18.483572477 +0000 UTC m=+35.999369289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:17.983698 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983512 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18bef297-7e7a-40d4-ac55-291487d73ee7-data-volume\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.983698 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983685 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00bae080-60c7-49c2-aff2-93e41d0e23be-config-volume\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983705 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983728 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00bae080-60c7-49c2-aff2-93e41d0e23be-tmp-dir\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983760 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18bef297-7e7a-40d4-ac55-291487d73ee7-data-volume\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:17.983814 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:17.983819 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:17.983863 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls podName:00bae080-60c7-49c2-aff2-93e41d0e23be nodeName:}" failed. No retries permitted until 2026-07-15 08:01:18.483849688 +0000 UTC m=+35.999646500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls") pod "dns-default-m8h8b" (UID: "00bae080-60c7-49c2-aff2-93e41d0e23be") : secret "dns-default-metrics-tls" not found Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983882 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bvb\" (UniqueName: \"kubernetes.io/projected/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-api-access-98bvb\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.983917 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18bef297-7e7a-40d4-ac55-291487d73ee7-crio-socket\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.984320 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:17.983933 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:18.483914223 +0000 UTC m=+35.999711038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : configmap "kube-rbac-proxy" not found Jul 15 08:01:17.984320 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.983969 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b07d8157-8401-4516-ac0d-20c2d5c65275-cert\") pod \"ingress-canary-x7gwv\" (UID: \"b07d8157-8401-4516-ac0d-20c2d5c65275\") " pod="openshift-ingress-canary/ingress-canary-x7gwv" Jul 15 08:01:17.984320 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.984001 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxpc\" (UniqueName: \"kubernetes.io/projected/00bae080-60c7-49c2-aff2-93e41d0e23be-kube-api-access-zlxpc\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.984320 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.984054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18bef297-7e7a-40d4-ac55-291487d73ee7-crio-socket\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:17.984320 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.984118 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00bae080-60c7-49c2-aff2-93e41d0e23be-tmp-dir\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.984320 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.984263 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00bae080-60c7-49c2-aff2-93e41d0e23be-config-volume\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:17.987824 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:17.987800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b07d8157-8401-4516-ac0d-20c2d5c65275-cert\") pod \"ingress-canary-x7gwv\" (UID: \"b07d8157-8401-4516-ac0d-20c2d5c65275\") " pod="openshift-ingress-canary/ingress-canary-x7gwv" Jul 15 08:01:18.000725 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.000662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n762h\" (UniqueName: \"kubernetes.io/projected/b07d8157-8401-4516-ac0d-20c2d5c65275-kube-api-access-n762h\") pod \"ingress-canary-x7gwv\" (UID: \"b07d8157-8401-4516-ac0d-20c2d5c65275\") " pod="openshift-ingress-canary/ingress-canary-x7gwv" Jul 15 08:01:18.022149 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.022125 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxpc\" (UniqueName: \"kubernetes.io/projected/00bae080-60c7-49c2-aff2-93e41d0e23be-kube-api-access-zlxpc\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:18.023666 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.023645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bvb\" (UniqueName: \"kubernetes.io/projected/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-api-access-98bvb\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:18.028588 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.028563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:18.028685 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.028595 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:18.033746 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.033727 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 08:01:18.033964 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.033947 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 08:01:18.034030 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.033984 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-df2lv\"" Jul 15 08:01:18.037107 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.037084 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:18.050191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.050074 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx"] Jul 15 08:01:18.052830 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.052813 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" Jul 15 08:01:18.055256 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.055238 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-scqjs\"" Jul 15 08:01:18.055395 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.055380 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 15 08:01:18.055624 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.055608 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 15 08:01:18.077118 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.077088 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx"] Jul 15 08:01:18.084363 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.084345 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfvg\" (UniqueName: \"kubernetes.io/projected/eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37-kube-api-access-cmfvg\") pod \"migrator-7f59b54f6c-mqshx\" (UID: \"eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" Jul 15 08:01:18.121579 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.121559 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7gwv" Jul 15 08:01:18.142068 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.142043 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:18.185294 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.185271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfvg\" (UniqueName: \"kubernetes.io/projected/eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37-kube-api-access-cmfvg\") pod \"migrator-7f59b54f6c-mqshx\" (UID: \"eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" Jul 15 08:01:18.194963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.194933 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfvg\" (UniqueName: \"kubernetes.io/projected/eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37-kube-api-access-cmfvg\") pod \"migrator-7f59b54f6c-mqshx\" (UID: \"eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" Jul 15 08:01:18.248498 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.248464 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:18.305511 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.305447 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-txzrm_7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2/dns-node-resolver/0.log" Jul 15 08:01:18.361952 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.361921 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" Jul 15 08:01:18.487664 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.487631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:18.487855 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.487699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:18.487855 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.487737 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:18.487855 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:18.487827 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:18.487855 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:18.487841 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:18.488053 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:18.487843 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:01:18.488053 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:18.487902 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:19.487882543 +0000 UTC m=+37.003679375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : configmap "kube-rbac-proxy" not found Jul 15 08:01:18.488053 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:18.487956 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:19.487937908 +0000 UTC m=+37.003734720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:18.488053 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:18.487984 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls podName:00bae080-60c7-49c2-aff2-93e41d0e23be nodeName:}" failed. No retries permitted until 2026-07-15 08:01:19.487971448 +0000 UTC m=+37.003768260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls") pod "dns-default-m8h8b" (UID: "00bae080-60c7-49c2-aff2-93e41d0e23be") : secret "dns-default-metrics-tls" not found Jul 15 08:01:18.915963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:18.915906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:19.016843 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.016792 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:19.028316 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.028288 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:19.032428 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.032404 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5x8qz\"" Jul 15 08:01:19.032979 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.032960 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 08:01:19.033074 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.033005 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 08:01:19.102390 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.102365 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-clz7b_451a8c3d-179c-4a38-a3c3-50c96043ebf1/node-ca/0.log" Jul 15 08:01:19.498706 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.498682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:19.499078 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.498731 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:19.499078 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.498756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:19.499078 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:19.498835 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:19.499078 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:19.498896 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:21.498874498 +0000 UTC m=+39.014671323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : configmap "kube-rbac-proxy" not found Jul 15 08:01:19.499078 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:19.498978 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:19.499078 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:19.499038 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:21.499022002 +0000 UTC m=+39.014818829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:19.499310 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:19.499113 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:01:19.499310 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:19.499149 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls podName:00bae080-60c7-49c2-aff2-93e41d0e23be nodeName:}" failed. No retries permitted until 2026-07-15 08:01:21.499137018 +0000 UTC m=+39.014933831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls") pod "dns-default-m8h8b" (UID: "00bae080-60c7-49c2-aff2-93e41d0e23be") : secret "dns-default-metrics-tls" not found Jul 15 08:01:19.661388 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.661223 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx"] Jul 15 08:01:19.685678 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:19.685583 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7gwv"] Jul 15 08:01:19.688542 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:19.688508 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3ce00e_fb28_4ef5_bdd2_46ac8e781c37.slice/crio-5b9ab5c41e0b1aa78762ba26c4e273cf1bbe6fabc85b386007df29e3c52031e6 WatchSource:0}: Error finding container 5b9ab5c41e0b1aa78762ba26c4e273cf1bbe6fabc85b386007df29e3c52031e6: Status 404 returned error can't find the container with id 5b9ab5c41e0b1aa78762ba26c4e273cf1bbe6fabc85b386007df29e3c52031e6 Jul 15 08:01:19.689512 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:19.689482 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07d8157_8401_4516_ac0d_20c2d5c65275.slice/crio-8b6930b17e3f2dee3250f9bf75ae2d91e4024af206e2ef26ec29eb920b9090b7 WatchSource:0}: Error finding container 8b6930b17e3f2dee3250f9bf75ae2d91e4024af206e2ef26ec29eb920b9090b7: Status 404 returned error can't find the container with id 8b6930b17e3f2dee3250f9bf75ae2d91e4024af206e2ef26ec29eb920b9090b7 Jul 15 08:01:20.255699 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:20.255659 2567 generic.go:356] "Generic (PLEG): container finished" podID="88e8e60f-c63e-43f8-9ac7-7cd0b3682693" containerID="84948082da19359580faa6d25a93c7604558d47f9d1a225f8d2664d09b5a63f1" exitCode=0 Jul 15 08:01:20.255882 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:20.255749 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47mlt" event={"ID":"88e8e60f-c63e-43f8-9ac7-7cd0b3682693","Type":"ContainerDied","Data":"84948082da19359580faa6d25a93c7604558d47f9d1a225f8d2664d09b5a63f1"} Jul 15 08:01:20.256839 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:20.256809 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7gwv" event={"ID":"b07d8157-8401-4516-ac0d-20c2d5c65275","Type":"ContainerStarted","Data":"8b6930b17e3f2dee3250f9bf75ae2d91e4024af206e2ef26ec29eb920b9090b7"} Jul 15 08:01:20.257853 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:20.257835 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" event={"ID":"eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37","Type":"ContainerStarted","Data":"5b9ab5c41e0b1aa78762ba26c4e273cf1bbe6fabc85b386007df29e3c52031e6"} Jul 15 08:01:20.503608 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:20.503575 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:21.264924 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:21.264886 2567 generic.go:356] "Generic (PLEG): container finished" podID="88e8e60f-c63e-43f8-9ac7-7cd0b3682693" containerID="fc6b9522b971ba904e0881937aac6c88eabed9cbc64c2ba3705bcadc4fee0baa" exitCode=0 Jul 15 08:01:21.265080 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:21.264951 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47mlt" event={"ID":"88e8e60f-c63e-43f8-9ac7-7cd0b3682693","Type":"ContainerDied","Data":"fc6b9522b971ba904e0881937aac6c88eabed9cbc64c2ba3705bcadc4fee0baa"} Jul 15 08:01:21.513705 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:21.513486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:21.514110 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:21.513736 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:21.514110 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:21.513657 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:01:21.514110 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:21.513822 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:21.514110 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:21.513840 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls podName:00bae080-60c7-49c2-aff2-93e41d0e23be nodeName:}" failed. No retries permitted until 2026-07-15 08:01:25.513816506 +0000 UTC m=+43.029613339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls") pod "dns-default-m8h8b" (UID: "00bae080-60c7-49c2-aff2-93e41d0e23be") : secret "dns-default-metrics-tls" not found Jul 15 08:01:21.514110 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:21.513910 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:21.514110 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:21.513916 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:21.514110 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:21.513960 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:25.51394496 +0000 UTC m=+43.029741790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:21.514110 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:21.513979 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:25.513969314 +0000 UTC m=+43.029766130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : configmap "kube-rbac-proxy" not found Jul 15 08:01:22.271981 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:22.271950 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47mlt" event={"ID":"88e8e60f-c63e-43f8-9ac7-7cd0b3682693","Type":"ContainerStarted","Data":"6f3b41f0583c8fa4fd63bcd59b2f63373c825847da5130980a7c9b3f2dfe41f3"} Jul 15 08:01:22.299160 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:22.299098 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-47mlt" podStartSLOduration=7.898146135 podStartE2EDuration="39.299075208s" totalImagesPullingTime="31.400929073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:22.298342956 +0000 UTC m=+39.814139802" watchObservedRunningTime="2026-07-15 08:01:22.299075208 +0000 UTC m=+39.814872042" Jul 15 08:01:22.981575 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:22.981552 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:23.223770 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:23.223729 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:23.276056 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:23.275977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7gwv" event={"ID":"b07d8157-8401-4516-ac0d-20c2d5c65275","Type":"ContainerStarted","Data":"a93999033f115df4ceeb10555d0423109ec2ddcbd6a5b723ad45fa25e6215b83"} Jul 15 08:01:23.277620 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:23.277598 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" event={"ID":"eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37","Type":"ContainerStarted","Data":"531c6a7546834618e3d57a6a0ec95e96ac46ce39863a8c86b2a2de0980c07561"} Jul 15 08:01:23.277719 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:23.277640 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" event={"ID":"eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37","Type":"ContainerStarted","Data":"9bfb9345bdbffa95f2e40fc23d5e810c0d01649811185ba375969fb68dd2ea0e"} Jul 15 08:01:23.294040 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:23.294001 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x7gwv" podStartSLOduration=3.050784791 podStartE2EDuration="6.293963922s" totalImagesPullingTime="3.243179131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:23.293175234 +0000 UTC m=+40.808972067" watchObservedRunningTime="2026-07-15 08:01:23.293963922 +0000 UTC m=+40.809760755" Jul 15 08:01:23.313182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:23.313073 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mqshx" podStartSLOduration=2.234677761 podStartE2EDuration="5.313061977s" totalImagesPullingTime="3.078384216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:23.312941176 +0000 UTC m=+40.828738020" watchObservedRunningTime="2026-07-15 08:01:23.313061977 +0000 UTC m=+40.828858808" Jul 15 08:01:23.329788 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:23.329760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:23.435507 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:23.435475 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:24.001572 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:24.001538 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:24.102747 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:24.102720 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:24.308174 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:24.308096 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:24.374007 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:24.373980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:24.913983 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:24.913947 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:25.114358 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:25.114328 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:25.318960 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:25.318878 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:25.546168 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:25.546126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:25.546363 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:25.546192 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:25.546363 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:25.546227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:25.546363 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:25.546265 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:25.546363 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:25.546319 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:01:25.546363 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:25.546346 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:33.546326338 +0000 UTC m=+51.062123150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : configmap "kube-rbac-proxy" not found Jul 15 08:01:25.546363 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:25.546323 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:25.546363 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:25.546366 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls podName:00bae080-60c7-49c2-aff2-93e41d0e23be nodeName:}" failed. No retries permitted until 2026-07-15 08:01:33.546356377 +0000 UTC m=+51.062153189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls") pod "dns-default-m8h8b" (UID: "00bae080-60c7-49c2-aff2-93e41d0e23be") : secret "dns-default-metrics-tls" not found Jul 15 08:01:25.546742 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:25.546381 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls podName:18bef297-7e7a-40d4-ac55-291487d73ee7 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:33.546372817 +0000 UTC m=+51.062169632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lnlh8" (UID: "18bef297-7e7a-40d4-ac55-291487d73ee7") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:26.147479 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:26.147423 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:26.517323 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:26.517247 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:27.194221 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:27.194192 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:27.270728 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:27.270704 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:27.344324 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:27.344296 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:27.727128 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:27.727094 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:27.786060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:27.786036 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:28.321246 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:28.321220 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:28.923651 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:28.923622 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:29.171695 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:29.171665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:29.317748 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:29.317683 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:29.379708 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:29.379689 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:29.579489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:29.579408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:29.713403 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:29.713378 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:30.324710 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:30.324684 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:30.446681 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:30.446654 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:30.530663 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:30.530636 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:30.931620 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:30.931593 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:31.126486 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:31.126459 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:31.339379 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:31.339307 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:31.652328 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:31.652299 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:31.754521 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:31.754487 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:33.608058 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.608024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:33.608425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.608065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:33.608425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.608115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:33.608709 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.608683 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18bef297-7e7a-40d4-ac55-291487d73ee7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:33.611804 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.611782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bae080-60c7-49c2-aff2-93e41d0e23be-metrics-tls\") pod \"dns-default-m8h8b\" (UID: \"00bae080-60c7-49c2-aff2-93e41d0e23be\") " pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:33.611890 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.611782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18bef297-7e7a-40d4-ac55-291487d73ee7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lnlh8\" (UID: \"18bef297-7e7a-40d4-ac55-291487d73ee7\") " pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:33.713578 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.713553 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:33.740233 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.740211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lnlh8" Jul 15 08:01:33.862265 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.862196 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m8h8b"] Jul 15 08:01:33.865132 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:33.865104 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bae080_60c7_49c2_aff2_93e41d0e23be.slice/crio-4ca8ef7737419935901218b1a46ad16243cd7208f9f101970c47e33df377656a WatchSource:0}: Error finding container 4ca8ef7737419935901218b1a46ad16243cd7208f9f101970c47e33df377656a: Status 404 returned error can't find the container with id 4ca8ef7737419935901218b1a46ad16243cd7208f9f101970c47e33df377656a Jul 15 08:01:33.884266 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:33.884242 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lnlh8"] Jul 15 08:01:33.887734 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:33.887711 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bef297_7e7a_40d4_ac55_291487d73ee7.slice/crio-a0bcb67cb925780e0aa731116f5f6240cb99224baeb786f4cc8a8f2845f9b631 WatchSource:0}: Error finding container a0bcb67cb925780e0aa731116f5f6240cb99224baeb786f4cc8a8f2845f9b631: Status 404 returned error can't find the container with id a0bcb67cb925780e0aa731116f5f6240cb99224baeb786f4cc8a8f2845f9b631 Jul 15 08:01:34.268888 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:34.268810 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:34.302988 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:34.302955 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lnlh8" event={"ID":"18bef297-7e7a-40d4-ac55-291487d73ee7","Type":"ContainerStarted","Data":"a0bcb67cb925780e0aa731116f5f6240cb99224baeb786f4cc8a8f2845f9b631"} Jul 15 08:01:34.304239 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:34.304210 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m8h8b" event={"ID":"00bae080-60c7-49c2-aff2-93e41d0e23be","Type":"ContainerStarted","Data":"4ca8ef7737419935901218b1a46ad16243cd7208f9f101970c47e33df377656a"} Jul 15 08:01:34.370830 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:34.370798 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:36.186208 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.186179 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:36.312897 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.312863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m8h8b" event={"ID":"00bae080-60c7-49c2-aff2-93e41d0e23be","Type":"ContainerStarted","Data":"2f77cc4e5b1c78cfbdd7214f64aaa12ce11c7757bdbbe66f727bd302f643fd24"} Jul 15 08:01:36.313154 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.312957 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m8h8b" event={"ID":"00bae080-60c7-49c2-aff2-93e41d0e23be","Type":"ContainerStarted","Data":"b51176cd0b7d20e6e2ac2c6d5949bc28705c4ac717e038a555c460c590fbfcb1"} Jul 15 08:01:36.313154 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.313006 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:36.338248 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.338194 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m8h8b" podStartSLOduration=17.287668479 podStartE2EDuration="19.338177854s" totalImagesPullingTime="2.050509375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:36.337616692 +0000 UTC m=+53.853413528" watchObservedRunningTime="2026-07-15 08:01:36.338177854 +0000 UTC m=+53.853974678" Jul 15 08:01:36.338636 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.338614 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:36.435155 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.435082 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:36.501767 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.501731 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:36.850580 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.850547 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:36.923505 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:36.923477 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:37.087166 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:37.087133 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:37.262738 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:37.262651 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:37.679959 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:37.679920 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:37.743023 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:37.742987 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:37.875771 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:37.875738 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:37.934785 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:37.934691 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:38.890859 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:38.890821 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:38.960940 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:38.960906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:40.284576 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.283515 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58"] Jul 15 08:01:40.286586 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.285726 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" Jul 15 08:01:40.288548 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.288253 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 08:01:40.288688 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.288647 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 08:01:40.289607 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.289394 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 15 08:01:40.289759 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.289735 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ksf9t\"" Jul 15 08:01:40.289868 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.289794 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 08:01:40.297323 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.296367 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6"] Jul 15 08:01:40.298691 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.298456 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.300352 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.300329 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58"] Jul 15 08:01:40.300666 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.300645 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 08:01:40.312661 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.312621 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6"] Jul 15 08:01:40.330887 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.330855 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lnlh8" event={"ID":"18bef297-7e7a-40d4-ac55-291487d73ee7","Type":"ContainerStarted","Data":"5b5c71cb6568618819a11aa951cb389b40009e7883e46b96b7463b773fb542ad"} Jul 15 08:01:40.362827 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.362802 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9t5r\" (UniqueName: \"kubernetes.io/projected/c3334baa-e5bb-4106-912b-12fd08a225c9-kube-api-access-m9t5r\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.362981 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.362850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3334baa-e5bb-4106-912b-12fd08a225c9-tmp\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.362981 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.362879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c3334baa-e5bb-4106-912b-12fd08a225c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.362981 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.362964 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6fdfaef-e3ca-43f3-909c-3127f35c126a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78f699fd4b-59k58\" (UID: \"a6fdfaef-e3ca-43f3-909c-3127f35c126a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" Jul 15 08:01:40.363131 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.362983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkh2l\" (UniqueName: \"kubernetes.io/projected/a6fdfaef-e3ca-43f3-909c-3127f35c126a-kube-api-access-rkh2l\") pod \"managed-serviceaccount-addon-agent-78f699fd4b-59k58\" (UID: \"a6fdfaef-e3ca-43f3-909c-3127f35c126a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" Jul 15 08:01:40.386952 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.386922 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-2274l"] Jul 15 08:01:40.388904 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.388877 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-2274l" Jul 15 08:01:40.391567 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.391542 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 08:01:40.391674 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.391641 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 08:01:40.391674 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.391653 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wgmnh\"" Jul 15 08:01:40.403757 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.403732 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-2274l"] Jul 15 08:01:40.464350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.464317 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6hw\" (UniqueName: \"kubernetes.io/projected/fd0a15dd-889a-4c64-a1bb-6f41a6fa9161-kube-api-access-hd6hw\") pod \"downloads-68c9cbb497-2274l\" (UID: \"fd0a15dd-889a-4c64-a1bb-6f41a6fa9161\") " pod="openshift-console/downloads-68c9cbb497-2274l" Jul 15 08:01:40.464552 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.464368 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9t5r\" (UniqueName: \"kubernetes.io/projected/c3334baa-e5bb-4106-912b-12fd08a225c9-kube-api-access-m9t5r\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.464552 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.464399 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3334baa-e5bb-4106-912b-12fd08a225c9-tmp\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.464552 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.464418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c3334baa-e5bb-4106-912b-12fd08a225c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.464552 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.464483 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6fdfaef-e3ca-43f3-909c-3127f35c126a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78f699fd4b-59k58\" (UID: \"a6fdfaef-e3ca-43f3-909c-3127f35c126a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" Jul 15 08:01:40.464552 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.464506 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkh2l\" (UniqueName: \"kubernetes.io/projected/a6fdfaef-e3ca-43f3-909c-3127f35c126a-kube-api-access-rkh2l\") pod \"managed-serviceaccount-addon-agent-78f699fd4b-59k58\" (UID: \"a6fdfaef-e3ca-43f3-909c-3127f35c126a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" Jul 15 08:01:40.464815 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.464791 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3334baa-e5bb-4106-912b-12fd08a225c9-tmp\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.467301 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.467278 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c3334baa-e5bb-4106-912b-12fd08a225c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.467412 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.467283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6fdfaef-e3ca-43f3-909c-3127f35c126a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78f699fd4b-59k58\" (UID: \"a6fdfaef-e3ca-43f3-909c-3127f35c126a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" Jul 15 08:01:40.473958 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.473930 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:40.476255 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.476230 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9t5r\" (UniqueName: \"kubernetes.io/projected/c3334baa-e5bb-4106-912b-12fd08a225c9-kube-api-access-m9t5r\") pod \"klusterlet-addon-workmgr-85f6556fcd-68tl6\" (UID: \"c3334baa-e5bb-4106-912b-12fd08a225c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.477858 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.477830 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkh2l\" (UniqueName: \"kubernetes.io/projected/a6fdfaef-e3ca-43f3-909c-3127f35c126a-kube-api-access-rkh2l\") pod \"managed-serviceaccount-addon-agent-78f699fd4b-59k58\" (UID: \"a6fdfaef-e3ca-43f3-909c-3127f35c126a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" Jul 15 08:01:40.489472 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.489449 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f4b9f5896-h7vrj"] Jul 15 08:01:40.491770 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.491754 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.495101 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.495070 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 08:01:40.495101 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.495080 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 08:01:40.495254 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.495079 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2wb89\"" Jul 15 08:01:40.495356 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.495326 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 08:01:40.501169 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.501151 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 08:01:40.507734 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.507712 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4b9f5896-h7vrj"] Jul 15 08:01:40.557570 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.557482 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:40.565555 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7779880-a9eb-4e40-8499-e65c92b6034d-trusted-ca\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.565702 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565558 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8p9\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-kube-api-access-wm8p9\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.565702 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7779880-a9eb-4e40-8499-e65c92b6034d-installation-pull-secrets\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.565702 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-bound-sa-token\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.565872 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565702 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7779880-a9eb-4e40-8499-e65c92b6034d-ca-trust-extracted\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.565872 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6hw\" (UniqueName: \"kubernetes.io/projected/fd0a15dd-889a-4c64-a1bb-6f41a6fa9161-kube-api-access-hd6hw\") pod \"downloads-68c9cbb497-2274l\" (UID: \"fd0a15dd-889a-4c64-a1bb-6f41a6fa9161\") " pod="openshift-console/downloads-68c9cbb497-2274l" Jul 15 08:01:40.565872 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7779880-a9eb-4e40-8499-e65c92b6034d-registry-certificates\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.566019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565889 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-registry-tls\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.566019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.565912 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7779880-a9eb-4e40-8499-e65c92b6034d-image-registry-private-configuration\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.578750 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.578714 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6hw\" (UniqueName: \"kubernetes.io/projected/fd0a15dd-889a-4c64-a1bb-6f41a6fa9161-kube-api-access-hd6hw\") pod \"downloads-68c9cbb497-2274l\" (UID: \"fd0a15dd-889a-4c64-a1bb-6f41a6fa9161\") " pod="openshift-console/downloads-68c9cbb497-2274l" Jul 15 08:01:40.609242 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.609215 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" Jul 15 08:01:40.620211 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.620186 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.666536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7779880-a9eb-4e40-8499-e65c92b6034d-registry-certificates\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.666590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-registry-tls\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.666622 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7779880-a9eb-4e40-8499-e65c92b6034d-image-registry-private-configuration\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.666645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7779880-a9eb-4e40-8499-e65c92b6034d-trusted-ca\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.666671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8p9\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-kube-api-access-wm8p9\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.666707 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7779880-a9eb-4e40-8499-e65c92b6034d-installation-pull-secrets\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.666767 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-bound-sa-token\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.666796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7779880-a9eb-4e40-8499-e65c92b6034d-ca-trust-extracted\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.667129 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:40.667930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.667251 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7779880-a9eb-4e40-8499-e65c92b6034d-ca-trust-extracted\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.668515 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.668279 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7779880-a9eb-4e40-8499-e65c92b6034d-trusted-ca\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.669198 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.669020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7779880-a9eb-4e40-8499-e65c92b6034d-registry-certificates\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.671494 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.671407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7779880-a9eb-4e40-8499-e65c92b6034d-installation-pull-secrets\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.672731 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.672694 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-registry-tls\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.673633 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.673611 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7779880-a9eb-4e40-8499-e65c92b6034d-image-registry-private-configuration\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.683299 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.683272 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8p9\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-kube-api-access-wm8p9\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.686256 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.686205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7779880-a9eb-4e40-8499-e65c92b6034d-bound-sa-token\") pod \"image-registry-7f4b9f5896-h7vrj\" (UID: \"a7779880-a9eb-4e40-8499-e65c92b6034d\") " pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.699211 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.699034 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-2274l" Jul 15 08:01:40.759046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.759011 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:40.766416 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.766292 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58"] Jul 15 08:01:40.770210 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:40.770169 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6fdfaef_e3ca_43f3_909c_3127f35c126a.slice/crio-43a702562fd8d782e294f00729e74a19d789229ecb4d783a7763c9d026acd9d7 WatchSource:0}: Error finding container 43a702562fd8d782e294f00729e74a19d789229ecb4d783a7763c9d026acd9d7: Status 404 returned error can't find the container with id 43a702562fd8d782e294f00729e74a19d789229ecb4d783a7763c9d026acd9d7 Jul 15 08:01:40.790730 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.790702 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6"] Jul 15 08:01:40.793059 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:40.793037 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3334baa_e5bb_4106_912b_12fd08a225c9.slice/crio-b101a93e0b22c61bc66027c62f858a94f2ae863d28af7e0b130674acf8ec076c WatchSource:0}: Error finding container b101a93e0b22c61bc66027c62f858a94f2ae863d28af7e0b130674acf8ec076c: Status 404 returned error can't find the container with id b101a93e0b22c61bc66027c62f858a94f2ae863d28af7e0b130674acf8ec076c Jul 15 08:01:40.802531 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.802506 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:40.861507 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.861477 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-2274l"] Jul 15 08:01:40.867212 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:40.867181 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0a15dd_889a_4c64_a1bb_6f41a6fa9161.slice/crio-b5292d38984e862c306e12bf0dfd85d2f01f74cc17c45c2ce68184b702612950 WatchSource:0}: Error finding container b5292d38984e862c306e12bf0dfd85d2f01f74cc17c45c2ce68184b702612950: Status 404 returned error can't find the container with id b5292d38984e862c306e12bf0dfd85d2f01f74cc17c45c2ce68184b702612950 Jul 15 08:01:40.948908 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:40.948872 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4b9f5896-h7vrj"] Jul 15 08:01:41.335485 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:41.335421 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" event={"ID":"c3334baa-e5bb-4106-912b-12fd08a225c9","Type":"ContainerStarted","Data":"b101a93e0b22c61bc66027c62f858a94f2ae863d28af7e0b130674acf8ec076c"} Jul 15 08:01:41.336705 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:41.336675 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" event={"ID":"a6fdfaef-e3ca-43f3-909c-3127f35c126a","Type":"ContainerStarted","Data":"43a702562fd8d782e294f00729e74a19d789229ecb4d783a7763c9d026acd9d7"} Jul 15 08:01:41.337742 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:41.337711 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-2274l" event={"ID":"fd0a15dd-889a-4c64-a1bb-6f41a6fa9161","Type":"ContainerStarted","Data":"b5292d38984e862c306e12bf0dfd85d2f01f74cc17c45c2ce68184b702612950"} Jul 15 08:01:42.114934 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:42.114899 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7779880_a9eb_4e40_8499_e65c92b6034d.slice/crio-dba9a791a5d6498242ea91b9fb4814941f113d0f3295a54392fda9767321d49b WatchSource:0}: Error finding container dba9a791a5d6498242ea91b9fb4814941f113d0f3295a54392fda9767321d49b: Status 404 returned error can't find the container with id dba9a791a5d6498242ea91b9fb4814941f113d0f3295a54392fda9767321d49b Jul 15 08:01:42.349490 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:42.349413 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" event={"ID":"a7779880-a9eb-4e40-8499-e65c92b6034d","Type":"ContainerStarted","Data":"dba9a791a5d6498242ea91b9fb4814941f113d0f3295a54392fda9767321d49b"} Jul 15 08:01:43.246993 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:43.246921 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnbv5" Jul 15 08:01:43.356224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:43.356183 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" event={"ID":"a7779880-a9eb-4e40-8499-e65c92b6034d","Type":"ContainerStarted","Data":"dfda4ffed646e0f728c8ececfaed9c827eb6cf7f2537e80c14fce58bceddd788"} Jul 15 08:01:43.356699 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:43.356639 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:01:43.359140 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:43.359104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lnlh8" event={"ID":"18bef297-7e7a-40d4-ac55-291487d73ee7","Type":"ContainerStarted","Data":"143d5cdc7abd7ee604b79a3a79aa01fde01c222085a0682823c77bac9d2973fb"} Jul 15 08:01:43.379305 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:43.379248 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" podStartSLOduration=3.379227129 podStartE2EDuration="3.379227129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:43.37740194 +0000 UTC m=+60.893198776" watchObservedRunningTime="2026-07-15 08:01:43.379227129 +0000 UTC m=+60.895023964" Jul 15 08:01:43.663213 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:43.663175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:43.734884 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:43.734845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:44.714995 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:44.714958 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:44.827740 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:44.827695 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:45.367695 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:45.367659 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" event={"ID":"a6fdfaef-e3ca-43f3-909c-3127f35c126a","Type":"ContainerStarted","Data":"ede577ac4eb895972df4c10af017b11ef9864c4e699c2564570a7654a54dfb66"} Jul 15 08:01:45.413568 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:45.413510 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78f699fd4b-59k58" podStartSLOduration=1.339448429 podStartE2EDuration="5.413488194s" totalImagesPullingTime="4.074039765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:45.412199937 +0000 UTC m=+62.927996773" watchObservedRunningTime="2026-07-15 08:01:45.413488194 +0000 UTC m=+62.929285029" Jul 15 08:01:46.319956 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:46.319913 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m8h8b" Jul 15 08:01:46.442139 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:46.442100 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:46.517613 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:46.517533 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:48.383322 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.383282 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lnlh8" event={"ID":"18bef297-7e7a-40d4-ac55-291487d73ee7","Type":"ContainerStarted","Data":"159f3546ddedd9e209289b967921e0db5bbab847b85d18d5c1b5a68e8a956743"} Jul 15 08:01:48.402371 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.402315 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lnlh8" podStartSLOduration=17.237852288 podStartE2EDuration="31.402295723s" totalImagesPullingTime="14.164443435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:01:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:48.401409244 +0000 UTC m=+65.917206089" watchObservedRunningTime="2026-07-15 08:01:48.402295723 +0000 UTC m=+65.918092558" Jul 15 08:01:48.740520 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.740484 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:48.743239 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.743214 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 08:01:48.754306 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.754282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8a42fee-4675-42df-8f1a-71cb407b62ad-original-pull-secret\") pod \"global-pull-secret-syncer-nlzkz\" (UID: \"e8a42fee-4675-42df-8f1a-71cb407b62ad\") " pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:48.841822 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.841793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:48.841968 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.841835 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:48.844905 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.844887 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 08:01:48.845000 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.844889 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 08:01:48.855140 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.855116 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88bb6b5a-617c-4e10-871a-0060bbb1cdf3-metrics-certs\") pod \"network-metrics-daemon-f6cf9\" (UID: \"88bb6b5a-617c-4e10-871a-0060bbb1cdf3\") " pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:48.855459 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.855431 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 08:01:48.866071 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.866040 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99kdf\" (UniqueName: \"kubernetes.io/projected/1318d214-3429-46bc-9a2b-bcbcab7e56d5-kube-api-access-99kdf\") pod \"network-check-target-7tfd8\" (UID: \"1318d214-3429-46bc-9a2b-bcbcab7e56d5\") " pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:48.943329 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.943307 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-df2lv\"" Jul 15 08:01:48.946205 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.946188 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlzkz" Jul 15 08:01:48.951007 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:48.950986 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6cf9" Jul 15 08:01:49.041753 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:49.041519 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5x8qz\"" Jul 15 08:01:49.048891 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:49.048868 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:01:49.105474 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:49.105362 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f6cf9"] Jul 15 08:01:49.110069 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:49.110036 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bb6b5a_617c_4e10_871a_0060bbb1cdf3.slice/crio-d24cdf806c8593ac56019fd654527c8f24e6d8d78cdf79487033ce11e20e4a3c WatchSource:0}: Error finding container d24cdf806c8593ac56019fd654527c8f24e6d8d78cdf79487033ce11e20e4a3c: Status 404 returned error can't find the container with id d24cdf806c8593ac56019fd654527c8f24e6d8d78cdf79487033ce11e20e4a3c Jul 15 08:01:49.122788 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:49.122743 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nlzkz"] Jul 15 08:01:49.127569 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:49.127539 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a42fee_4675_42df_8f1a_71cb407b62ad.slice/crio-a08f68e5e48329f91423745236330411409dad4fc96c449c4bcaab5ac79fecaa WatchSource:0}: Error finding container a08f68e5e48329f91423745236330411409dad4fc96c449c4bcaab5ac79fecaa: Status 404 returned error can't find the container with id a08f68e5e48329f91423745236330411409dad4fc96c449c4bcaab5ac79fecaa Jul 15 08:01:49.206131 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:49.206103 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7tfd8"] Jul 15 08:01:49.209021 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:01:49.208968 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1318d214_3429_46bc_9a2b_bcbcab7e56d5.slice/crio-4432fc49fd157e068266cfc1e72e4b27a17069cb95808db597e2291dd51bc9ff WatchSource:0}: Error finding container 4432fc49fd157e068266cfc1e72e4b27a17069cb95808db597e2291dd51bc9ff: Status 404 returned error can't find the container with id 4432fc49fd157e068266cfc1e72e4b27a17069cb95808db597e2291dd51bc9ff Jul 15 08:01:49.388263 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:49.388223 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6cf9" event={"ID":"88bb6b5a-617c-4e10-871a-0060bbb1cdf3","Type":"ContainerStarted","Data":"d24cdf806c8593ac56019fd654527c8f24e6d8d78cdf79487033ce11e20e4a3c"} Jul 15 08:01:49.389549 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:49.389521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7tfd8" event={"ID":"1318d214-3429-46bc-9a2b-bcbcab7e56d5","Type":"ContainerStarted","Data":"4432fc49fd157e068266cfc1e72e4b27a17069cb95808db597e2291dd51bc9ff"} Jul 15 08:01:49.390704 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:49.390680 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nlzkz" event={"ID":"e8a42fee-4675-42df-8f1a-71cb407b62ad","Type":"ContainerStarted","Data":"a08f68e5e48329f91423745236330411409dad4fc96c449c4bcaab5ac79fecaa"} Jul 15 08:01:50.914034 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:50.913219 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:51.015160 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:51.015070 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:51.402019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:51.401922 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6cf9" event={"ID":"88bb6b5a-617c-4e10-871a-0060bbb1cdf3","Type":"ContainerStarted","Data":"4853815b63b8a64dde0c410c21fef42677095d56fa1f813bc119fcc0cd2cd18c"} Jul 15 08:01:51.402019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:51.401989 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6cf9" event={"ID":"88bb6b5a-617c-4e10-871a-0060bbb1cdf3","Type":"ContainerStarted","Data":"495f38d1783f45c2382f71f10bbd0f590a007e951466d6d4f36bbbd8e5cf9978"} Jul 15 08:01:51.422624 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:51.422571 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f6cf9" podStartSLOduration=66.736510117 podStartE2EDuration="1m8.42255123s" totalImagesPullingTime="1.686041113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:51.422012658 +0000 UTC m=+68.937809493" watchObservedRunningTime="2026-07-15 08:01:51.42255123 +0000 UTC m=+68.938348064" Jul 15 08:01:51.553304 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:51.553272 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:51.655064 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:51.654946 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:55.220424 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:55.220384 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:57.150321 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.150275 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:57.411808 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.411564 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4"] Jul 15 08:01:57.445428 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.445398 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8rrq7"] Jul 15 08:01:57.445618 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.445528 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.448178 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.448148 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 08:01:57.448389 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.448362 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 08:01:57.448536 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.448358 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d8rjp\"" Jul 15 08:01:57.448990 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.448968 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 08:01:57.449081 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.449052 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 08:01:57.449141 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.448970 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 08:01:57.450030 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.450010 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4"] Jul 15 08:01:57.450151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.450130 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.451034 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.450978 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8rrq7"] Jul 15 08:01:57.452928 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.452894 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 08:01:57.453019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.452985 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 08:01:57.453171 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.453106 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 08:01:57.453328 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.453309 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-456sk\"" Jul 15 08:01:57.458801 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.458673 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7rs4n"] Jul 15 08:01:57.471081 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.471056 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.473907 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.473887 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 08:01:57.474026 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.473893 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 08:01:57.474091 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.473916 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fqpfw\"" Jul 15 08:01:57.474164 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.473935 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 08:01:57.512529 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.512503 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79574c66c7-czc6z"] Jul 15 08:01:57.515707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.515686 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.518926 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.518907 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kcpks\"" Jul 15 08:01:57.519026 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.519003 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 08:01:57.519259 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.519241 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 08:01:57.519907 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.519866 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 08:01:57.520005 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.519930 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 08:01:57.520093 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.520002 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 08:01:57.526398 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.526381 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 08:01:57.530024 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.530001 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79574c66c7-czc6z"] Jul 15 08:01:57.556892 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.556848 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:57.609573 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609540 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-service-ca\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.609728 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609587 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-oauth-config\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.609728 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609616 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-oauth-serving-cert\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.609728 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609679 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-root\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.609728 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-wtmp\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.609910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxpv\" (UniqueName: \"kubernetes.io/projected/a1a83d63-7ef9-4d16-be23-cc1375e6822a-kube-api-access-ftxpv\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.609910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609784 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-metrics-client-ca\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.609910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609808 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-sys\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.609910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-textfile\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.609910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609890 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.609910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609909 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pkn\" (UniqueName: \"kubernetes.io/projected/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-kube-api-access-m4pkn\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.610191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609953 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08366be9-0b75-46cf-b941-d181ee2c8b51-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.610191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.609985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6htp\" (UniqueName: \"kubernetes.io/projected/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-api-access-c6htp\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.610191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.610191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610052 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.610191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610079 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a83d63-7ef9-4d16-be23-cc1375e6822a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.610191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkwf\" (UniqueName: \"kubernetes.io/projected/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-kube-api-access-glkwf\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.610191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-serving-cert\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.610503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610198 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.610503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.610503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610254 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-tls\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.610503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610279 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-trusted-ca-bundle\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.610503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-config\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.610503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610342 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.610503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/08366be9-0b75-46cf-b941-d181ee2c8b51-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.610503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.610413 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a83d63-7ef9-4d16-be23-cc1375e6822a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711535 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkwf\" (UniqueName: \"kubernetes.io/projected/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-kube-api-access-glkwf\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-serving-cert\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-tls\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711665 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-trusted-ca-bundle\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711691 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-config\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/08366be9-0b75-46cf-b941-d181ee2c8b51-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711784 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711817 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-service-ca\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711869 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-oauth-config\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-oauth-serving-cert\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.712425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-root\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711946 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-wtmp\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.711983 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxpv\" (UniqueName: \"kubernetes.io/projected/a1a83d63-7ef9-4d16-be23-cc1375e6822a-kube-api-access-ftxpv\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712015 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-metrics-client-ca\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-sys\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712064 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-textfile\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pkn\" (UniqueName: \"kubernetes.io/projected/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-kube-api-access-m4pkn\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712145 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08366be9-0b75-46cf-b941-d181ee2c8b51-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6htp\" (UniqueName: \"kubernetes.io/projected/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-api-access-c6htp\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712205 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712523 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-root\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712714 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a83d63-7ef9-4d16-be23-cc1375e6822a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712831 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-textfile\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.712920 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-sys\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.713419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.713051 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-wtmp\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.714198 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.713471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/08366be9-0b75-46cf-b941-d181ee2c8b51-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.714198 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.714123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.714340 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08366be9-0b75-46cf-b941-d181ee2c8b51-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.714417 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-trusted-ca-bundle\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:57.714534 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:01:57.714598 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-tls podName:a1a83d63-7ef9-4d16-be23-cc1375e6822a nodeName:}" failed. No retries permitted until 2026-07-15 08:01:58.21457728 +0000 UTC m=+75.730374095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-ndxf4" (UID: "a1a83d63-7ef9-4d16-be23-cc1375e6822a") : secret "openshift-state-metrics-tls" not found Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.714947 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-service-ca\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.714948 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-config\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.715141 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-oauth-serving-cert\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.715359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.715452 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.715391 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-metrics-client-ca\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.715889 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.715779 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.716863 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.716837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-serving-cert\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.717085 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.717064 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.717605 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.717581 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-oauth-config\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.718068 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.718044 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-node-exporter-tls\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.719385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.719366 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.721931 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.721908 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkwf\" (UniqueName: \"kubernetes.io/projected/e15d3bdc-32c7-4041-9d8c-c38459b2a1a1-kube-api-access-glkwf\") pod \"node-exporter-7rs4n\" (UID: \"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1\") " pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.725924 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.725897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxpv\" (UniqueName: \"kubernetes.io/projected/a1a83d63-7ef9-4d16-be23-cc1375e6822a-kube-api-access-ftxpv\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:57.726488 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.726463 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pkn\" (UniqueName: \"kubernetes.io/projected/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-kube-api-access-m4pkn\") pod \"console-79574c66c7-czc6z\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.726776 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.726753 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6htp\" (UniqueName: \"kubernetes.io/projected/08366be9-0b75-46cf-b941-d181ee2c8b51-kube-api-access-c6htp\") pod \"kube-state-metrics-785778ff74-8rrq7\" (UID: \"08366be9-0b75-46cf-b941-d181ee2c8b51\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.768912 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.768888 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" Jul 15 08:01:57.782103 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.782076 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7rs4n" Jul 15 08:01:57.827778 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.827707 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:01:57.879021 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.878990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:57.979633 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:57.979525 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:58.079665 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:58.079627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:58.183496 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:58.183463 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:58.216666 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:58.216631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:58.219506 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:58.219481 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1a83d63-7ef9-4d16-be23-cc1375e6822a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-ndxf4\" (UID: \"a1a83d63-7ef9-4d16-be23-cc1375e6822a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:58.359145 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:58.359117 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" Jul 15 08:01:58.360201 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:58.360174 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:58.445760 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:58.445726 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:58.672204 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:58.672129 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:59.669071 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:59.669029 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:59.728809 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:01:59.728771 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:00.396539 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:00.396466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:00.462603 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:00.462562 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:00.678851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:00.678768 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:00.784176 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:00.784133 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:00.880872 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:00.880822 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:01.140430 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.140400 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:01.813889 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.813822 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6567f9cb8b-lgvbl"] Jul 15 08:02:01.835313 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.835284 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6567f9cb8b-lgvbl"] Jul 15 08:02:01.835500 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.835376 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:01.839166 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.839131 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 08:02:01.839286 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.839181 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 08:02:01.839286 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.839132 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9qk48plu18vm0\"" Jul 15 08:02:01.839286 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.839131 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 08:02:01.839286 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.839143 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 08:02:01.839286 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.839285 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bcvjf\"" Jul 15 08:02:01.874634 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.874606 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:01.945030 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.944997 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:01.952084 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.952062 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xc9f\" (UniqueName: \"kubernetes.io/projected/069b1964-cf1e-457f-bc61-7b9ba4b383da-kube-api-access-9xc9f\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:01.952195 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.952101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-client-ca-bundle\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:01.952195 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.952125 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/069b1964-cf1e-457f-bc61-7b9ba4b383da-metrics-server-audit-profiles\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:01.952280 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.952196 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-secret-metrics-server-tls\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:01.952280 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.952229 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-secret-metrics-server-client-certs\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:01.952280 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.952250 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/069b1964-cf1e-457f-bc61-7b9ba4b383da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:01.952408 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:01.952282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/069b1964-cf1e-457f-bc61-7b9ba4b383da-audit-log\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.053452 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.053410 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xc9f\" (UniqueName: \"kubernetes.io/projected/069b1964-cf1e-457f-bc61-7b9ba4b383da-kube-api-access-9xc9f\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.053573 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.053471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-client-ca-bundle\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.053573 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.053504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/069b1964-cf1e-457f-bc61-7b9ba4b383da-metrics-server-audit-profiles\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.053573 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.053536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-secret-metrics-server-tls\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.053573 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.053561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-secret-metrics-server-client-certs\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.053761 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.053589 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/069b1964-cf1e-457f-bc61-7b9ba4b383da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.053761 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.053629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/069b1964-cf1e-457f-bc61-7b9ba4b383da-audit-log\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.054045 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.054014 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/069b1964-cf1e-457f-bc61-7b9ba4b383da-audit-log\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.054309 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.054284 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/069b1964-cf1e-457f-bc61-7b9ba4b383da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.054516 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.054497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/069b1964-cf1e-457f-bc61-7b9ba4b383da-metrics-server-audit-profiles\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.055987 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.055965 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-secret-metrics-server-tls\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.056083 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.056011 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-client-ca-bundle\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.056130 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.056078 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/069b1964-cf1e-457f-bc61-7b9ba4b383da-secret-metrics-server-client-certs\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.062350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.062304 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xc9f\" (UniqueName: \"kubernetes.io/projected/069b1964-cf1e-457f-bc61-7b9ba4b383da-kube-api-access-9xc9f\") pod \"metrics-server-6567f9cb8b-lgvbl\" (UID: \"069b1964-cf1e-457f-bc61-7b9ba4b383da\") " pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.145059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.145025 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:02.249538 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:02:02.249499 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15d3bdc_32c7_4041_9d8c_c38459b2a1a1.slice/crio-547b0babb98fad7590cee86202e3bf58fcfef27806f8cdf8b8af60e11665001d WatchSource:0}: Error finding container 547b0babb98fad7590cee86202e3bf58fcfef27806f8cdf8b8af60e11665001d: Status 404 returned error can't find the container with id 547b0babb98fad7590cee86202e3bf58fcfef27806f8cdf8b8af60e11665001d Jul 15 08:02:02.298640 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.298604 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:02.384925 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.384704 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:02.452562 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.452472 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6567f9cb8b-lgvbl"] Jul 15 08:02:02.464077 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.464006 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7tfd8" event={"ID":"1318d214-3429-46bc-9a2b-bcbcab7e56d5","Type":"ContainerStarted","Data":"9845a05d7d62c592e323913b5364581d6b6d1bd0c839750c8d36733a5e3189bf"} Jul 15 08:02:02.464189 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.464104 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:02:02.467769 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.467617 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:02.467769 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.467618 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rs4n" event={"ID":"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1","Type":"ContainerStarted","Data":"547b0babb98fad7590cee86202e3bf58fcfef27806f8cdf8b8af60e11665001d"} Jul 15 08:02:02.483966 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.482471 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4"] Jul 15 08:02:02.486849 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:02:02.486817 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a83d63_7ef9_4d16_be23_cc1375e6822a.slice/crio-dadf7f4cc85c706a1f962537a6c120eb062a866ba03ac84e87a11a615eadc4ca WatchSource:0}: Error finding container dadf7f4cc85c706a1f962537a6c120eb062a866ba03ac84e87a11a615eadc4ca: Status 404 returned error can't find the container with id dadf7f4cc85c706a1f962537a6c120eb062a866ba03ac84e87a11a615eadc4ca Jul 15 08:02:02.507304 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.506776 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7tfd8" podStartSLOduration=66.475828124 podStartE2EDuration="1m19.506754429s" totalImagesPullingTime="13.030926305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:02.485420257 +0000 UTC m=+80.001217092" watchObservedRunningTime="2026-07-15 08:02:02.506754429 +0000 UTC m=+80.022551264" Jul 15 08:02:02.507304 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.507233 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8rrq7"] Jul 15 08:02:02.512534 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:02:02.512505 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08366be9_0b75_46cf_b941_d181ee2c8b51.slice/crio-41f3710895d58f9ee67974f02afda19edc319f9ca264d45b072c57a3ff4a3690 WatchSource:0}: Error finding container 41f3710895d58f9ee67974f02afda19edc319f9ca264d45b072c57a3ff4a3690: Status 404 returned error can't find the container with id 41f3710895d58f9ee67974f02afda19edc319f9ca264d45b072c57a3ff4a3690 Jul 15 08:02:02.717427 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:02.717305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79574c66c7-czc6z"] Jul 15 08:02:02.720622 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:02:02.720594 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea46cd8_a53b_4f08_a92e_0bd8da5883b1.slice/crio-4e25e1afa0bf020038a0db6bada9089147daa43f22e4686b7eda828bac721b86 WatchSource:0}: Error finding container 4e25e1afa0bf020038a0db6bada9089147daa43f22e4686b7eda828bac721b86: Status 404 returned error can't find the container with id 4e25e1afa0bf020038a0db6bada9089147daa43f22e4686b7eda828bac721b86 Jul 15 08:02:03.479997 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.479534 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nlzkz" event={"ID":"e8a42fee-4675-42df-8f1a-71cb407b62ad","Type":"ContainerStarted","Data":"89cc76f2fbd35afad4261794eaa96d94b0eaa29e4ef5ddc861c897a7f997425f"} Jul 15 08:02:03.486887 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.486855 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-2274l" event={"ID":"fd0a15dd-889a-4c64-a1bb-6f41a6fa9161","Type":"ContainerStarted","Data":"ba484a024e040411a468906aa654da731c48890fc55badcc293825f609f7457f"} Jul 15 08:02:03.487651 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.487628 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-2274l" Jul 15 08:02:03.491630 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.491605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" event={"ID":"069b1964-cf1e-457f-bc61-7b9ba4b383da","Type":"ContainerStarted","Data":"ac60d7b30587b6fc2f5877cf15d2229d9de2e772fdba7a1097667f3545b480d3"} Jul 15 08:02:03.497856 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.497803 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nlzkz" podStartSLOduration=67.385532821 podStartE2EDuration="1m20.497785867s" totalImagesPullingTime="13.112253046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:03.497033953 +0000 UTC m=+81.012830790" watchObservedRunningTime="2026-07-15 08:02:03.497785867 +0000 UTC m=+81.013582702" Jul 15 08:02:03.499188 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.499159 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79574c66c7-czc6z" event={"ID":"bea46cd8-a53b-4f08-a92e-0bd8da5883b1","Type":"ContainerStarted","Data":"4e25e1afa0bf020038a0db6bada9089147daa43f22e4686b7eda828bac721b86"} Jul 15 08:02:03.504833 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.504805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" event={"ID":"a1a83d63-7ef9-4d16-be23-cc1375e6822a","Type":"ContainerStarted","Data":"895375dd97c959bcc6e92c071be9eeff7cf1629fe592ec3bbad5eac83776109e"} Jul 15 08:02:03.504931 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.504866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" event={"ID":"a1a83d63-7ef9-4d16-be23-cc1375e6822a","Type":"ContainerStarted","Data":"fbf71761f48c557c4523a7f6ba752420f18fbf437c4ecd6d01703ec133cc2eaf"} Jul 15 08:02:03.504931 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.504894 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" event={"ID":"a1a83d63-7ef9-4d16-be23-cc1375e6822a","Type":"ContainerStarted","Data":"dadf7f4cc85c706a1f962537a6c120eb062a866ba03ac84e87a11a615eadc4ca"} Jul 15 08:02:03.507099 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.507035 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" event={"ID":"08366be9-0b75-46cf-b941-d181ee2c8b51","Type":"ContainerStarted","Data":"41f3710895d58f9ee67974f02afda19edc319f9ca264d45b072c57a3ff4a3690"} Jul 15 08:02:03.508922 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.508899 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-2274l" Jul 15 08:02:03.517540 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.517480 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-2274l" podStartSLOduration=2.086250018 podStartE2EDuration="23.517461832s" totalImagesPullingTime="21.431211814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:03.514843908 +0000 UTC m=+81.030640743" watchObservedRunningTime="2026-07-15 08:02:03.517461832 +0000 UTC m=+81.033258670" Jul 15 08:02:03.671363 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:03.671269 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:04.010474 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:04.010410 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:04.111504 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:04.111369 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:04.306356 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:04.306240 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79574c66c7-czc6z"] Jul 15 08:02:04.369291 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:04.369250 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f4b9f5896-h7vrj" Jul 15 08:02:04.536757 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:04.536656 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:04.598985 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:04.598896 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:10.547638 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.547600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" event={"ID":"069b1964-cf1e-457f-bc61-7b9ba4b383da","Type":"ContainerStarted","Data":"1cc19584a19c3ee0fe1bc8095312a8f989365685926a7480537b247ab1c209e4"} Jul 15 08:02:10.549719 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.549680 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79574c66c7-czc6z" event={"ID":"bea46cd8-a53b-4f08-a92e-0bd8da5883b1","Type":"ContainerStarted","Data":"cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245"} Jul 15 08:02:10.551776 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.551744 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" event={"ID":"c3334baa-e5bb-4106-912b-12fd08a225c9","Type":"ContainerStarted","Data":"856eb42fdc6cb4f8d10abb17de1f7a98e8ec5e7eebe1949f987ffd50c7aae87a"} Jul 15 08:02:10.551933 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.551918 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:02:10.554513 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.554488 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" event={"ID":"a1a83d63-7ef9-4d16-be23-cc1375e6822a","Type":"ContainerStarted","Data":"85cbad99003acff11334731e0e00f1e12603008b3235b63d92e6a17cdcba743b"} Jul 15 08:02:10.555897 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.555878 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" Jul 15 08:02:10.557380 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.557355 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" event={"ID":"08366be9-0b75-46cf-b941-d181ee2c8b51","Type":"ContainerStarted","Data":"82d471ccf5656cbf1973f42e6d059ae4a6908d40e6c5da6a636c90b3090cc9c7"} Jul 15 08:02:10.557496 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.557405 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" event={"ID":"08366be9-0b75-46cf-b941-d181ee2c8b51","Type":"ContainerStarted","Data":"f046e05eb00a1ae98611a02f70a214ac0e7e265c4bb45bc51a318c4b0ccd0a11"} Jul 15 08:02:10.557496 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.557453 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" event={"ID":"08366be9-0b75-46cf-b941-d181ee2c8b51","Type":"ContainerStarted","Data":"b5139d96afad4fef593adea0c8e2edee29573a8cde26aabf16d7a6328cdf918c"} Jul 15 08:02:10.559958 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.559917 2567 generic.go:356] "Generic (PLEG): container finished" podID="e15d3bdc-32c7-4041-9d8c-c38459b2a1a1" containerID="3452bd2becde8bf01bd93a393dc1fd0918e1ae020ed8a8afad5c8c9c39586690" exitCode=0 Jul 15 08:02:10.560051 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.559988 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rs4n" event={"ID":"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1","Type":"ContainerDied","Data":"3452bd2becde8bf01bd93a393dc1fd0918e1ae020ed8a8afad5c8c9c39586690"} Jul 15 08:02:10.565499 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.565455 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" podStartSLOduration=2.366405921 podStartE2EDuration="9.565424652s" totalImagesPullingTime="7.199018731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:10.564494118 +0000 UTC m=+88.080290959" watchObservedRunningTime="2026-07-15 08:02:10.565424652 +0000 UTC m=+88.081221464" Jul 15 08:02:10.589561 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.586422 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-8rrq7" podStartSLOduration=6.441143545 podStartE2EDuration="13.58640322s" totalImagesPullingTime="7.145259675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:10.585189781 +0000 UTC m=+88.100986616" watchObservedRunningTime="2026-07-15 08:02:10.58640322 +0000 UTC m=+88.102200055" Jul 15 08:02:10.607957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.607906 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f6556fcd-68tl6" podStartSLOduration=1.702147074 podStartE2EDuration="30.60789222s" totalImagesPullingTime="28.905745146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:10.60607187 +0000 UTC m=+88.121868705" watchObservedRunningTime="2026-07-15 08:02:10.60789222 +0000 UTC m=+88.123689054" Jul 15 08:02:10.628034 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.627993 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ndxf4" podStartSLOduration=6.601638316 podStartE2EDuration="13.62797461s" totalImagesPullingTime="7.026336294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:10.62588396 +0000 UTC m=+88.141680795" watchObservedRunningTime="2026-07-15 08:02:10.62797461 +0000 UTC m=+88.143771446" Jul 15 08:02:10.664075 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:10.664023 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79574c66c7-czc6z" podStartSLOduration=6.677855705 podStartE2EDuration="13.664002199s" totalImagesPullingTime="6.986146494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:10.662716534 +0000 UTC m=+88.178513369" watchObservedRunningTime="2026-07-15 08:02:10.664002199 +0000 UTC m=+88.179799034" Jul 15 08:02:11.566017 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:11.565979 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rs4n" event={"ID":"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1","Type":"ContainerStarted","Data":"1410517771c7893f1da7f6dabfd62bc963496a1149d01b47e7fd0459458e080c"} Jul 15 08:02:11.566475 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:11.566038 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rs4n" event={"ID":"e15d3bdc-32c7-4041-9d8c-c38459b2a1a1","Type":"ContainerStarted","Data":"1999cf974d68fdf42780b61b1bfb2324dc91d0db3f6532a0a033febb3b501b2d"} Jul 15 08:02:11.589965 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:11.589915 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7rs4n" podStartSLOduration=8.835733819 podStartE2EDuration="14.589899466s" totalImagesPullingTime="5.59463264s" totalInitContainerRuntime="159.533007ms" isStatefulPod=true podCreationTimestamp="2026-07-15 08:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:11.588009836 +0000 UTC m=+89.103806669" watchObservedRunningTime="2026-07-15 08:02:11.589899466 +0000 UTC m=+89.105696343" Jul 15 08:02:13.217475 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:13.217417 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:13.298822 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:13.298794 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:15.446056 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:15.446024 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:15.552066 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:15.552035 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:17.828234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:17.828202 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:02:19.044053 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:19.044014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:19.349498 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:19.349471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:19.449888 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:19.449861 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:22.145538 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:22.145507 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:22.145905 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:22.145565 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:23.411282 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:23.411242 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:23.511489 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:23.511458 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:23.582547 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:23.582517 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:30.426809 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:30.426778 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:30.491309 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:30.491287 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:33.509393 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:33.509361 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7tfd8" Jul 15 08:02:35.580763 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.580704 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79574c66c7-czc6z" podUID="bea46cd8-a53b-4f08-a92e-0bd8da5883b1" containerName="console" containerID="cri-o://cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245" gracePeriod=15 Jul 15 08:02:35.845347 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.845325 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:02:35.919566 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.919538 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:35.923582 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.923563 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-trusted-ca-bundle\") pod \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " Jul 15 08:02:35.923654 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.923597 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-oauth-config\") pod \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " Jul 15 08:02:35.923654 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.923619 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-oauth-serving-cert\") pod \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " Jul 15 08:02:35.923654 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.923649 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-config\") pod \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " Jul 15 08:02:35.923800 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.923717 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-service-ca\") pod \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " Jul 15 08:02:35.923800 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.923751 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4pkn\" (UniqueName: \"kubernetes.io/projected/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-kube-api-access-m4pkn\") pod \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " Jul 15 08:02:35.923800 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.923790 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-serving-cert\") pod \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\" (UID: \"bea46cd8-a53b-4f08-a92e-0bd8da5883b1\") " Jul 15 08:02:35.924109 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.924074 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bea46cd8-a53b-4f08-a92e-0bd8da5883b1" (UID: "bea46cd8-a53b-4f08-a92e-0bd8da5883b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:35.924109 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.924087 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bea46cd8-a53b-4f08-a92e-0bd8da5883b1" (UID: "bea46cd8-a53b-4f08-a92e-0bd8da5883b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:35.924222 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.924172 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "bea46cd8-a53b-4f08-a92e-0bd8da5883b1" (UID: "bea46cd8-a53b-4f08-a92e-0bd8da5883b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:35.924222 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.924202 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-config" (OuterVolumeSpecName: "console-config") pod "bea46cd8-a53b-4f08-a92e-0bd8da5883b1" (UID: "bea46cd8-a53b-4f08-a92e-0bd8da5883b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:35.925818 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.925791 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bea46cd8-a53b-4f08-a92e-0bd8da5883b1" (UID: "bea46cd8-a53b-4f08-a92e-0bd8da5883b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:02:35.925977 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.925958 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bea46cd8-a53b-4f08-a92e-0bd8da5883b1" (UID: "bea46cd8-a53b-4f08-a92e-0bd8da5883b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:02:35.926023 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.925996 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-kube-api-access-m4pkn" (OuterVolumeSpecName: "kube-api-access-m4pkn") pod "bea46cd8-a53b-4f08-a92e-0bd8da5883b1" (UID: "bea46cd8-a53b-4f08-a92e-0bd8da5883b1"). InnerVolumeSpecName "kube-api-access-m4pkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:02:35.987292 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:35.987269 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:02:36.025299 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.025279 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:02:36.025399 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.025304 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4pkn\" (UniqueName: \"kubernetes.io/projected/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-kube-api-access-m4pkn\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:02:36.025399 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.025313 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-serving-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:02:36.025399 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.025322 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-trusted-ca-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:02:36.025399 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.025331 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-oauth-config\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:02:36.025399 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.025340 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-oauth-serving-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:02:36.025399 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.025347 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bea46cd8-a53b-4f08-a92e-0bd8da5883b1-console-config\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:02:36.656877 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.656838 2567 generic.go:356] "Generic (PLEG): container finished" podID="bea46cd8-a53b-4f08-a92e-0bd8da5883b1" containerID="cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245" exitCode=0 Jul 15 08:02:36.657316 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.656883 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79574c66c7-czc6z" Jul 15 08:02:36.657316 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.656912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79574c66c7-czc6z" event={"ID":"bea46cd8-a53b-4f08-a92e-0bd8da5883b1","Type":"ContainerDied","Data":"cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245"} Jul 15 08:02:36.657316 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.656958 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79574c66c7-czc6z" event={"ID":"bea46cd8-a53b-4f08-a92e-0bd8da5883b1","Type":"ContainerDied","Data":"4e25e1afa0bf020038a0db6bada9089147daa43f22e4686b7eda828bac721b86"} Jul 15 08:02:36.657316 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.656986 2567 scope.go:117] "RemoveContainer" containerID="cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245" Jul 15 08:02:36.673860 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.673841 2567 scope.go:117] "RemoveContainer" containerID="cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245" Jul 15 08:02:36.674109 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:02:36.674087 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245\": container with ID starting with cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245 not found: ID does not exist" containerID="cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245" Jul 15 08:02:36.674202 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.674117 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245"} err="failed to get container status \"cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245\": rpc error: code = NotFound desc = could not find container \"cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245\": container with ID starting with cc162f13a5bf2dd1aae6710b03400880f501fa0c1f6720f519db1e9404b79245 not found: ID does not exist" Jul 15 08:02:36.677786 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.677765 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79574c66c7-czc6z"] Jul 15 08:02:36.681727 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:36.681707 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79574c66c7-czc6z"] Jul 15 08:02:37.032646 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:37.032578 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bea46cd8-a53b-4f08-a92e-0bd8da5883b1" path="/var/lib/kubelet/pods/bea46cd8-a53b-4f08-a92e-0bd8da5883b1/volumes" Jul 15 08:02:42.150496 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:42.150467 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:02:42.154364 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:02:42.154342 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6567f9cb8b-lgvbl" Jul 15 08:03:13.675060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:13.675024 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:13.798927 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:13.798903 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:18.159386 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:18.159344 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:25.674110 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:25.674079 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:25.800386 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:25.800353 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:27.860861 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:27.860827 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:28.034229 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:28.034192 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:28.169820 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:28.169711 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:28.534493 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:28.534396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:28.637274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:28.637241 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:30.455333 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:30.455304 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:31.174374 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:31.174341 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:38.308631 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:38.308559 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:38.409331 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:38.409299 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:38.493595 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:38.493574 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:03:44.093121 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:03:44.093092 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:05.632206 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:05.632174 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:05.732730 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:05.732699 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:25.573314 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.573281 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt"] Jul 15 08:04:25.573699 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.573580 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea46cd8-a53b-4f08-a92e-0bd8da5883b1" containerName="console" Jul 15 08:04:25.573699 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.573591 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea46cd8-a53b-4f08-a92e-0bd8da5883b1" containerName="console" Jul 15 08:04:25.573699 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.573628 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea46cd8-a53b-4f08-a92e-0bd8da5883b1" containerName="console" Jul 15 08:04:25.575619 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.575599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.579375 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.578897 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wxdmf\"" Jul 15 08:04:25.579375 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.579203 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 08:04:25.580678 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.579825 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 08:04:25.584941 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.584921 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt"] Jul 15 08:04:25.682185 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.682165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.682273 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.682189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.682273 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.682207 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrl4s\" (UniqueName: \"kubernetes.io/projected/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-kube-api-access-lrl4s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.783417 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.783393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.783516 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.783421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.783516 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.783456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrl4s\" (UniqueName: \"kubernetes.io/projected/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-kube-api-access-lrl4s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.783765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.783744 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.783799 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.783778 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.792896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.792876 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrl4s\" (UniqueName: \"kubernetes.io/projected/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-kube-api-access-lrl4s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:25.888472 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:25.888434 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:26.009507 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:26.009478 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt"] Jul 15 08:04:26.012573 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:04:26.012546 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bc0632_24e4_4bf4_9d71_8f2f8c75eb07.slice/crio-3feca8df1065c4c7a7f64579b3ea576ee27bc90f7530fce9ed0d6cbf20617131 WatchSource:0}: Error finding container 3feca8df1065c4c7a7f64579b3ea576ee27bc90f7530fce9ed0d6cbf20617131: Status 404 returned error can't find the container with id 3feca8df1065c4c7a7f64579b3ea576ee27bc90f7530fce9ed0d6cbf20617131 Jul 15 08:04:26.032495 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:26.032467 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" event={"ID":"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07","Type":"ContainerStarted","Data":"3feca8df1065c4c7a7f64579b3ea576ee27bc90f7530fce9ed0d6cbf20617131"} Jul 15 08:04:33.061589 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:33.061554 2567 generic.go:356] "Generic (PLEG): container finished" podID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerID="91a3176a23741279edb4385abca4ef3bdfa4adc1e6c319ce2c4309c9d460f0be" exitCode=0 Jul 15 08:04:33.061927 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:33.061605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" event={"ID":"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07","Type":"ContainerDied","Data":"91a3176a23741279edb4385abca4ef3bdfa4adc1e6c319ce2c4309c9d460f0be"} Jul 15 08:04:35.070011 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:35.069985 2567 generic.go:356] "Generic (PLEG): container finished" podID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerID="3f767f59567c6afeb5dfa5ea78312ea248509a6289d597d46116e8438215fb04" exitCode=0 Jul 15 08:04:35.070325 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:35.070025 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" event={"ID":"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07","Type":"ContainerDied","Data":"3f767f59567c6afeb5dfa5ea78312ea248509a6289d597d46116e8438215fb04"} Jul 15 08:04:43.102963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:43.102929 2567 generic.go:356] "Generic (PLEG): container finished" podID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerID="d1cf3776527c785694caf57eff09e51b8926196ff0e020b1305d1aeaeb6df265" exitCode=0 Jul 15 08:04:43.103372 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:43.102977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" event={"ID":"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07","Type":"ContainerDied","Data":"d1cf3776527c785694caf57eff09e51b8926196ff0e020b1305d1aeaeb6df265"} Jul 15 08:04:44.227009 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.226987 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:44.325515 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.325493 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-util\") pod \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " Jul 15 08:04:44.325623 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.325532 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrl4s\" (UniqueName: \"kubernetes.io/projected/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-kube-api-access-lrl4s\") pod \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " Jul 15 08:04:44.325623 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.325562 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-bundle\") pod \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\" (UID: \"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07\") " Jul 15 08:04:44.326229 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.326203 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-bundle" (OuterVolumeSpecName: "bundle") pod "88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" (UID: "88bc0632-24e4-4bf4-9d71-8f2f8c75eb07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:04:44.327898 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.327873 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-kube-api-access-lrl4s" (OuterVolumeSpecName: "kube-api-access-lrl4s") pod "88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" (UID: "88bc0632-24e4-4bf4-9d71-8f2f8c75eb07"). InnerVolumeSpecName "kube-api-access-lrl4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:04:44.329963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.329944 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-util" (OuterVolumeSpecName: "util") pod "88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" (UID: "88bc0632-24e4-4bf4-9d71-8f2f8c75eb07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:04:44.426431 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.426372 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:04:44.426431 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.426399 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-util\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:04:44.426431 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:44.426409 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrl4s\" (UniqueName: \"kubernetes.io/projected/88bc0632-24e4-4bf4-9d71-8f2f8c75eb07-kube-api-access-lrl4s\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:04:45.111552 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:45.111524 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" event={"ID":"88bc0632-24e4-4bf4-9d71-8f2f8c75eb07","Type":"ContainerDied","Data":"3feca8df1065c4c7a7f64579b3ea576ee27bc90f7530fce9ed0d6cbf20617131"} Jul 15 08:04:45.111552 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:45.111553 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cnzzt" Jul 15 08:04:45.111715 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:45.111570 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3feca8df1065c4c7a7f64579b3ea576ee27bc90f7530fce9ed0d6cbf20617131" Jul 15 08:04:46.594819 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:46.594776 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:47.553685 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:47.553652 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:48.543829 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.543790 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw"] Jul 15 08:04:48.544190 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.544045 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerName="pull" Jul 15 08:04:48.544190 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.544055 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerName="pull" Jul 15 08:04:48.544190 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.544064 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerName="util" Jul 15 08:04:48.544190 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.544069 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerName="util" Jul 15 08:04:48.544190 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.544079 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerName="extract" Jul 15 08:04:48.544190 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.544084 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerName="extract" Jul 15 08:04:48.544190 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.544125 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bc0632-24e4-4bf4-9d71-8f2f8c75eb07" containerName="extract" Jul 15 08:04:48.585338 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.585310 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw"] Jul 15 08:04:48.585505 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.585370 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" Jul 15 08:04:48.587938 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.587916 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:04:48.588061 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.587918 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-d65zz\"" Jul 15 08:04:48.588061 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.587927 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 15 08:04:48.756230 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.756206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r94p\" (UniqueName: \"kubernetes.io/projected/4f434783-a809-47c4-b325-27974cfe5def-kube-api-access-2r94p\") pod \"cert-manager-operator-controller-manager-55c444bf86-db7jw\" (UID: \"4f434783-a809-47c4-b325-27974cfe5def\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" Jul 15 08:04:48.756342 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.756240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f434783-a809-47c4-b325-27974cfe5def-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-db7jw\" (UID: \"4f434783-a809-47c4-b325-27974cfe5def\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" Jul 15 08:04:48.856517 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.856486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r94p\" (UniqueName: \"kubernetes.io/projected/4f434783-a809-47c4-b325-27974cfe5def-kube-api-access-2r94p\") pod \"cert-manager-operator-controller-manager-55c444bf86-db7jw\" (UID: \"4f434783-a809-47c4-b325-27974cfe5def\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" Jul 15 08:04:48.856636 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.856531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f434783-a809-47c4-b325-27974cfe5def-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-db7jw\" (UID: \"4f434783-a809-47c4-b325-27974cfe5def\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" Jul 15 08:04:48.856869 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.856853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f434783-a809-47c4-b325-27974cfe5def-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-db7jw\" (UID: \"4f434783-a809-47c4-b325-27974cfe5def\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" Jul 15 08:04:48.868191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.868170 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r94p\" (UniqueName: \"kubernetes.io/projected/4f434783-a809-47c4-b325-27974cfe5def-kube-api-access-2r94p\") pod \"cert-manager-operator-controller-manager-55c444bf86-db7jw\" (UID: \"4f434783-a809-47c4-b325-27974cfe5def\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" Jul 15 08:04:48.894002 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:48.893981 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" Jul 15 08:04:49.067312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:49.067278 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw"] Jul 15 08:04:49.071837 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:04:49.071813 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f434783_a809_47c4_b325_27974cfe5def.slice/crio-232e4d72ea8edd5e593b9b0a5a6cbb1796757d837d452299c5430136b6a5f5a3 WatchSource:0}: Error finding container 232e4d72ea8edd5e593b9b0a5a6cbb1796757d837d452299c5430136b6a5f5a3: Status 404 returned error can't find the container with id 232e4d72ea8edd5e593b9b0a5a6cbb1796757d837d452299c5430136b6a5f5a3 Jul 15 08:04:49.126347 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:49.126294 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" event={"ID":"4f434783-a809-47c4-b325-27974cfe5def","Type":"ContainerStarted","Data":"232e4d72ea8edd5e593b9b0a5a6cbb1796757d837d452299c5430136b6a5f5a3"} Jul 15 08:04:50.269164 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:50.269130 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:50.365498 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:50.365460 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:52.138847 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.138814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" event={"ID":"4f434783-a809-47c4-b325-27974cfe5def","Type":"ContainerStarted","Data":"b5d6cf0e9b63de85b23ce496e9987a315377fa21adf2ae3116b0ebe7c45376f1"} Jul 15 08:04:52.166274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.166231 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-db7jw" podStartSLOduration=1.9519583649999999 podStartE2EDuration="4.166210601s" totalImagesPullingTime="2.214252236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:04:52.163952968 +0000 UTC m=+249.679749802" watchObservedRunningTime="2026-07-15 08:04:52.166210601 +0000 UTC m=+249.682007439" Jul 15 08:04:52.168137 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.168115 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:52.270098 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.270076 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:52.365659 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.365634 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xdm78"] Jul 15 08:04:52.384690 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.384670 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xdm78"] Jul 15 08:04:52.384807 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.384749 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:52.387171 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.387150 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6468fd5-e7be-4c6a-8dfe-410db655e691-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xdm78\" (UID: \"b6468fd5-e7be-4c6a-8dfe-410db655e691\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:52.387266 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.387193 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdxx\" (UniqueName: \"kubernetes.io/projected/b6468fd5-e7be-4c6a-8dfe-410db655e691-kube-api-access-zhdxx\") pod \"cert-manager-webhook-6486ffd997-xdm78\" (UID: \"b6468fd5-e7be-4c6a-8dfe-410db655e691\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:52.391046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.390995 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-982rc\"" Jul 15 08:04:52.391147 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.391087 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 08:04:52.391600 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.391580 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 08:04:52.487752 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.487726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdxx\" (UniqueName: \"kubernetes.io/projected/b6468fd5-e7be-4c6a-8dfe-410db655e691-kube-api-access-zhdxx\") pod \"cert-manager-webhook-6486ffd997-xdm78\" (UID: \"b6468fd5-e7be-4c6a-8dfe-410db655e691\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:52.487862 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.487772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6468fd5-e7be-4c6a-8dfe-410db655e691-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xdm78\" (UID: \"b6468fd5-e7be-4c6a-8dfe-410db655e691\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:52.500961 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.500939 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6468fd5-e7be-4c6a-8dfe-410db655e691-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xdm78\" (UID: \"b6468fd5-e7be-4c6a-8dfe-410db655e691\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:52.501047 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.501034 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdxx\" (UniqueName: \"kubernetes.io/projected/b6468fd5-e7be-4c6a-8dfe-410db655e691-kube-api-access-zhdxx\") pod \"cert-manager-webhook-6486ffd997-xdm78\" (UID: \"b6468fd5-e7be-4c6a-8dfe-410db655e691\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:52.666318 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.666233 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:52.693744 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.693721 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:52.801790 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.801761 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:52.849692 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.849669 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xdm78"] Jul 15 08:04:52.851310 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:04:52.851282 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6468fd5_e7be_4c6a_8dfe_410db655e691.slice/crio-f939f93f9ea4e0a1185a02f8508005a57dcdc5464d1af3608369036a53483375 WatchSource:0}: Error finding container f939f93f9ea4e0a1185a02f8508005a57dcdc5464d1af3608369036a53483375: Status 404 returned error can't find the container with id f939f93f9ea4e0a1185a02f8508005a57dcdc5464d1af3608369036a53483375 Jul 15 08:04:52.939231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.939177 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2"] Jul 15 08:04:52.942241 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.942226 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:52.945659 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.945637 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wxdmf\"" Jul 15 08:04:52.945761 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.945640 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 08:04:52.948414 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.948392 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 08:04:52.957556 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.957534 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2"] Jul 15 08:04:52.990799 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.990766 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmwdz\" (UniqueName: \"kubernetes.io/projected/5b936678-f926-4911-9a66-e89ad1a223ee-kube-api-access-bmwdz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:52.990910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.990820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:52.990910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:52.990898 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:53.091942 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.091912 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmwdz\" (UniqueName: \"kubernetes.io/projected/5b936678-f926-4911-9a66-e89ad1a223ee-kube-api-access-bmwdz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:53.092075 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.091990 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:53.092075 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.092027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:53.092343 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.092323 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:53.092408 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.092359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:53.110243 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.110215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmwdz\" (UniqueName: \"kubernetes.io/projected/5b936678-f926-4911-9a66-e89ad1a223ee-kube-api-access-bmwdz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:53.143304 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.143276 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" event={"ID":"b6468fd5-e7be-4c6a-8dfe-410db655e691","Type":"ContainerStarted","Data":"f939f93f9ea4e0a1185a02f8508005a57dcdc5464d1af3608369036a53483375"} Jul 15 08:04:53.252131 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.252080 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:53.382309 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:53.382208 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2"] Jul 15 08:04:53.384782 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:04:53.384756 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b936678_f926_4911_9a66_e89ad1a223ee.slice/crio-237ab5ad465dabf6364a5e9de14e9b83dbdae02e70ae8e70d3c919cac8961897 WatchSource:0}: Error finding container 237ab5ad465dabf6364a5e9de14e9b83dbdae02e70ae8e70d3c919cac8961897: Status 404 returned error can't find the container with id 237ab5ad465dabf6364a5e9de14e9b83dbdae02e70ae8e70d3c919cac8961897 Jul 15 08:04:54.021987 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:54.021927 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:54.150134 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:54.150098 2567 generic.go:356] "Generic (PLEG): container finished" podID="5b936678-f926-4911-9a66-e89ad1a223ee" containerID="e3d191a19c0f91e8a5777599055de06d192ef3f4144fc52d1daae6d955d0553f" exitCode=0 Jul 15 08:04:54.150565 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:54.150203 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" event={"ID":"5b936678-f926-4911-9a66-e89ad1a223ee","Type":"ContainerDied","Data":"e3d191a19c0f91e8a5777599055de06d192ef3f4144fc52d1daae6d955d0553f"} Jul 15 08:04:54.150565 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:54.150267 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" event={"ID":"5b936678-f926-4911-9a66-e89ad1a223ee","Type":"ContainerStarted","Data":"237ab5ad465dabf6364a5e9de14e9b83dbdae02e70ae8e70d3c919cac8961897"} Jul 15 08:04:54.340998 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:54.340916 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:54.414166 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:54.414130 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:54.569274 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:54.569137 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:57.164513 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:57.164479 2567 generic.go:356] "Generic (PLEG): container finished" podID="5b936678-f926-4911-9a66-e89ad1a223ee" containerID="d722543f49cab0e1fb194f2d084d11c50a203bd91815a6918b2e031143ef0189" exitCode=0 Jul 15 08:04:57.164900 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:57.164571 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" event={"ID":"5b936678-f926-4911-9a66-e89ad1a223ee","Type":"ContainerDied","Data":"d722543f49cab0e1fb194f2d084d11c50a203bd91815a6918b2e031143ef0189"} Jul 15 08:04:57.166120 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:57.166100 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" event={"ID":"b6468fd5-e7be-4c6a-8dfe-410db655e691","Type":"ContainerStarted","Data":"4607648d035bdb5fba3ece541240638001fa3704f55994eda9069ea28d3650ca"} Jul 15 08:04:57.166262 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:57.166249 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:04:57.206724 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:57.206678 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" podStartSLOduration=1.721370547 podStartE2EDuration="5.2066643s" totalImagesPullingTime="3.485293753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:04:57.205510391 +0000 UTC m=+254.721307226" watchObservedRunningTime="2026-07-15 08:04:57.2066643 +0000 UTC m=+254.722461133" Jul 15 08:04:57.220191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:57.220169 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:57.319007 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:57.318973 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:58.171681 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:58.171651 2567 generic.go:356] "Generic (PLEG): container finished" podID="5b936678-f926-4911-9a66-e89ad1a223ee" containerID="0dd81e143c6eb6c89111c85afdeada47c65d3113882534ad29f0a17c50f203e3" exitCode=0 Jul 15 08:04:58.172048 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:58.171733 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" event={"ID":"5b936678-f926-4911-9a66-e89ad1a223ee","Type":"ContainerDied","Data":"0dd81e143c6eb6c89111c85afdeada47c65d3113882534ad29f0a17c50f203e3"} Jul 15 08:04:59.297759 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.297738 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:04:59.339490 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.339466 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-util\") pod \"5b936678-f926-4911-9a66-e89ad1a223ee\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " Jul 15 08:04:59.339595 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.339510 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmwdz\" (UniqueName: \"kubernetes.io/projected/5b936678-f926-4911-9a66-e89ad1a223ee-kube-api-access-bmwdz\") pod \"5b936678-f926-4911-9a66-e89ad1a223ee\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " Jul 15 08:04:59.339595 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.339586 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-bundle\") pod \"5b936678-f926-4911-9a66-e89ad1a223ee\" (UID: \"5b936678-f926-4911-9a66-e89ad1a223ee\") " Jul 15 08:04:59.339936 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.339910 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-bundle" (OuterVolumeSpecName: "bundle") pod "5b936678-f926-4911-9a66-e89ad1a223ee" (UID: "5b936678-f926-4911-9a66-e89ad1a223ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:04:59.341474 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.341430 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b936678-f926-4911-9a66-e89ad1a223ee-kube-api-access-bmwdz" (OuterVolumeSpecName: "kube-api-access-bmwdz") pod "5b936678-f926-4911-9a66-e89ad1a223ee" (UID: "5b936678-f926-4911-9a66-e89ad1a223ee"). InnerVolumeSpecName "kube-api-access-bmwdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:04:59.346940 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.346918 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-util" (OuterVolumeSpecName: "util") pod "5b936678-f926-4911-9a66-e89ad1a223ee" (UID: "5b936678-f926-4911-9a66-e89ad1a223ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:04:59.441019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.440973 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:04:59.441019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.440990 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b936678-f926-4911-9a66-e89ad1a223ee-util\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:04:59.441019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.440999 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmwdz\" (UniqueName: \"kubernetes.io/projected/5b936678-f926-4911-9a66-e89ad1a223ee-kube-api-access-bmwdz\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:04:59.860112 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.860083 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:04:59.945141 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:04:59.945108 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:00.181428 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:00.181352 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" event={"ID":"5b936678-f926-4911-9a66-e89ad1a223ee","Type":"ContainerDied","Data":"237ab5ad465dabf6364a5e9de14e9b83dbdae02e70ae8e70d3c919cac8961897"} Jul 15 08:05:00.181428 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:00.181393 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="237ab5ad465dabf6364a5e9de14e9b83dbdae02e70ae8e70d3c919cac8961897" Jul 15 08:05:00.181428 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:00.181365 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp7bl2" Jul 15 08:05:03.173837 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:03.173808 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xdm78" Jul 15 08:05:03.378889 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:03.378857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:08.849463 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:08.849366 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:08.941490 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:08.941458 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:15.297105 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:15.297073 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:15.398345 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:15.398318 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:17.149367 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:17.149334 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:18.528906 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.528872 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2"] Jul 15 08:05:18.529234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.529136 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b936678-f926-4911-9a66-e89ad1a223ee" containerName="util" Jul 15 08:05:18.529234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.529146 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b936678-f926-4911-9a66-e89ad1a223ee" containerName="util" Jul 15 08:05:18.529234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.529156 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b936678-f926-4911-9a66-e89ad1a223ee" containerName="pull" Jul 15 08:05:18.529234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.529162 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b936678-f926-4911-9a66-e89ad1a223ee" containerName="pull" Jul 15 08:05:18.529234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.529171 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b936678-f926-4911-9a66-e89ad1a223ee" containerName="extract" Jul 15 08:05:18.529234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.529177 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b936678-f926-4911-9a66-e89ad1a223ee" containerName="extract" Jul 15 08:05:18.529234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.529219 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b936678-f926-4911-9a66-e89ad1a223ee" containerName="extract" Jul 15 08:05:18.531195 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.531179 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.533705 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.533684 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 08:05:18.533793 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.533745 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 08:05:18.534688 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.534669 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wxdmf\"" Jul 15 08:05:18.540468 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.540431 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2"] Jul 15 08:05:18.574028 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.573997 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.574159 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.574076 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldp5w\" (UniqueName: \"kubernetes.io/projected/a650377e-c45b-4d96-a785-eac125a03910-kube-api-access-ldp5w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.574159 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.574113 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.674815 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.674789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldp5w\" (UniqueName: \"kubernetes.io/projected/a650377e-c45b-4d96-a785-eac125a03910-kube-api-access-ldp5w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.674905 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.674823 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.675035 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.675012 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.675234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.675218 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.675364 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.675282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.682886 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.682868 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldp5w\" (UniqueName: \"kubernetes.io/projected/a650377e-c45b-4d96-a785-eac125a03910-kube-api-access-ldp5w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.840918 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.840822 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:18.968679 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:18.968654 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2"] Jul 15 08:05:18.970921 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:05:18.970891 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda650377e_c45b_4d96_a785_eac125a03910.slice/crio-4054ad2f9aad674013eaaabf0c4ab63d517e1d84b262af5ef03ea2c1acade581 WatchSource:0}: Error finding container 4054ad2f9aad674013eaaabf0c4ab63d517e1d84b262af5ef03ea2c1acade581: Status 404 returned error can't find the container with id 4054ad2f9aad674013eaaabf0c4ab63d517e1d84b262af5ef03ea2c1acade581 Jul 15 08:05:19.258852 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:19.258814 2567 generic.go:356] "Generic (PLEG): container finished" podID="a650377e-c45b-4d96-a785-eac125a03910" containerID="6200c74f96f45020713fcd1233d7e72129c4a17ec7520f5196dfce5a6f890432" exitCode=0 Jul 15 08:05:19.259010 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:19.258853 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" event={"ID":"a650377e-c45b-4d96-a785-eac125a03910","Type":"ContainerDied","Data":"6200c74f96f45020713fcd1233d7e72129c4a17ec7520f5196dfce5a6f890432"} Jul 15 08:05:19.259010 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:19.258899 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" event={"ID":"a650377e-c45b-4d96-a785-eac125a03910","Type":"ContainerStarted","Data":"4054ad2f9aad674013eaaabf0c4ab63d517e1d84b262af5ef03ea2c1acade581"} Jul 15 08:05:20.264498 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:20.264394 2567 generic.go:356] "Generic (PLEG): container finished" podID="a650377e-c45b-4d96-a785-eac125a03910" containerID="aea03e440bd4d1fe2bf311eeac8124ac9944aaa3fa626afdf7080d8cb432837e" exitCode=0 Jul 15 08:05:20.264498 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:20.264474 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" event={"ID":"a650377e-c45b-4d96-a785-eac125a03910","Type":"ContainerDied","Data":"aea03e440bd4d1fe2bf311eeac8124ac9944aaa3fa626afdf7080d8cb432837e"} Jul 15 08:05:21.270056 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.270026 2567 generic.go:356] "Generic (PLEG): container finished" podID="a650377e-c45b-4d96-a785-eac125a03910" containerID="14b3ec28c0eb03e509bb83b204bb67e99dc584d5aaa9c3b5597e6a65ed64d04c" exitCode=0 Jul 15 08:05:21.270475 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.270086 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" event={"ID":"a650377e-c45b-4d96-a785-eac125a03910","Type":"ContainerDied","Data":"14b3ec28c0eb03e509bb83b204bb67e99dc584d5aaa9c3b5597e6a65ed64d04c"} Jul 15 08:05:21.363576 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.363542 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:21.436958 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.436925 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt"] Jul 15 08:05:21.439896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.439880 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.442758 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.442738 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 08:05:21.443018 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.442998 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:05:21.443119 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.443018 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 08:05:21.443119 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.443021 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 08:05:21.443225 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.443210 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wtg45\"" Jul 15 08:05:21.443475 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.443460 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 08:05:21.452142 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.452120 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt"] Jul 15 08:05:21.452870 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.452839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:21.495425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.495399 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d924e4e8-08db-4342-bf5a-efeea9fe23bb-metrics-cert\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.495425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.495433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d924e4e8-08db-4342-bf5a-efeea9fe23bb-manager-config\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.495635 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.495474 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ktm9\" (UniqueName: \"kubernetes.io/projected/d924e4e8-08db-4342-bf5a-efeea9fe23bb-kube-api-access-2ktm9\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.495635 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.495564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d924e4e8-08db-4342-bf5a-efeea9fe23bb-cert\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.596380 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.596354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d924e4e8-08db-4342-bf5a-efeea9fe23bb-cert\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.596535 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.596428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d924e4e8-08db-4342-bf5a-efeea9fe23bb-metrics-cert\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.596535 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.596481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d924e4e8-08db-4342-bf5a-efeea9fe23bb-manager-config\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.596535 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.596509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ktm9\" (UniqueName: \"kubernetes.io/projected/d924e4e8-08db-4342-bf5a-efeea9fe23bb-kube-api-access-2ktm9\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.597082 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.597058 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d924e4e8-08db-4342-bf5a-efeea9fe23bb-manager-config\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.598771 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.598747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d924e4e8-08db-4342-bf5a-efeea9fe23bb-metrics-cert\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.598919 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.598900 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d924e4e8-08db-4342-bf5a-efeea9fe23bb-cert\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.606604 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.606580 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ktm9\" (UniqueName: \"kubernetes.io/projected/d924e4e8-08db-4342-bf5a-efeea9fe23bb-kube-api-access-2ktm9\") pod \"lws-controller-manager-5bdbd9fc98-722zt\" (UID: \"d924e4e8-08db-4342-bf5a-efeea9fe23bb\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.748769 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.748744 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:21.873949 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:21.873869 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt"] Jul 15 08:05:21.876824 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:05:21.876798 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd924e4e8_08db_4342_bf5a_efeea9fe23bb.slice/crio-ee62eb2b5b22a72a6d7be74044a8e706ce842c8c3349a5fe4a8ef997394670a8 WatchSource:0}: Error finding container ee62eb2b5b22a72a6d7be74044a8e706ce842c8c3349a5fe4a8ef997394670a8: Status 404 returned error can't find the container with id ee62eb2b5b22a72a6d7be74044a8e706ce842c8c3349a5fe4a8ef997394670a8 Jul 15 08:05:22.274986 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.274947 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" event={"ID":"d924e4e8-08db-4342-bf5a-efeea9fe23bb","Type":"ContainerStarted","Data":"ee62eb2b5b22a72a6d7be74044a8e706ce842c8c3349a5fe4a8ef997394670a8"} Jul 15 08:05:22.389783 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.389759 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:22.506819 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.506750 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-util\") pod \"a650377e-c45b-4d96-a785-eac125a03910\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " Jul 15 08:05:22.506819 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.506817 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-bundle\") pod \"a650377e-c45b-4d96-a785-eac125a03910\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " Jul 15 08:05:22.506992 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.506851 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldp5w\" (UniqueName: \"kubernetes.io/projected/a650377e-c45b-4d96-a785-eac125a03910-kube-api-access-ldp5w\") pod \"a650377e-c45b-4d96-a785-eac125a03910\" (UID: \"a650377e-c45b-4d96-a785-eac125a03910\") " Jul 15 08:05:22.507538 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.507512 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-bundle" (OuterVolumeSpecName: "bundle") pod "a650377e-c45b-4d96-a785-eac125a03910" (UID: "a650377e-c45b-4d96-a785-eac125a03910"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:05:22.508852 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.508830 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a650377e-c45b-4d96-a785-eac125a03910-kube-api-access-ldp5w" (OuterVolumeSpecName: "kube-api-access-ldp5w") pod "a650377e-c45b-4d96-a785-eac125a03910" (UID: "a650377e-c45b-4d96-a785-eac125a03910"). InnerVolumeSpecName "kube-api-access-ldp5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:05:22.512573 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.512552 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-util" (OuterVolumeSpecName: "util") pod "a650377e-c45b-4d96-a785-eac125a03910" (UID: "a650377e-c45b-4d96-a785-eac125a03910"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:05:22.607693 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.607673 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldp5w\" (UniqueName: \"kubernetes.io/projected/a650377e-c45b-4d96-a785-eac125a03910-kube-api-access-ldp5w\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:05:22.607774 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.607695 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-util\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:05:22.607774 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:22.607703 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a650377e-c45b-4d96-a785-eac125a03910-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:05:23.281852 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:23.281811 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" event={"ID":"a650377e-c45b-4d96-a785-eac125a03910","Type":"ContainerDied","Data":"4054ad2f9aad674013eaaabf0c4ab63d517e1d84b262af5ef03ea2c1acade581"} Jul 15 08:05:23.281852 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:23.281852 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzcbr2" Jul 15 08:05:23.282355 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:23.281866 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4054ad2f9aad674013eaaabf0c4ab63d517e1d84b262af5ef03ea2c1acade581" Jul 15 08:05:24.287429 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:24.287397 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" event={"ID":"d924e4e8-08db-4342-bf5a-efeea9fe23bb","Type":"ContainerStarted","Data":"fad16f74279e85fea2deaf7cee5d69418d48140463716699e47fdace1541bacd"} Jul 15 08:05:24.287916 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:24.287543 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:24.306384 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:24.306338 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" podStartSLOduration=1.491403718 podStartE2EDuration="3.306323889s" totalImagesPullingTime="1.814920171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:05:24.305514156 +0000 UTC m=+281.821310989" watchObservedRunningTime="2026-07-15 08:05:24.306323889 +0000 UTC m=+281.822120723" Jul 15 08:05:30.280228 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:30.280195 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:30.379613 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:30.379576 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:30.822015 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:30.821965 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:31.342197 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.342167 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:31.438368 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.438339 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:31.487867 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.487843 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s"] Jul 15 08:05:31.488115 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.488102 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a650377e-c45b-4d96-a785-eac125a03910" containerName="pull" Jul 15 08:05:31.488156 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.488116 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a650377e-c45b-4d96-a785-eac125a03910" containerName="pull" Jul 15 08:05:31.488156 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.488126 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a650377e-c45b-4d96-a785-eac125a03910" containerName="util" Jul 15 08:05:31.488156 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.488131 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a650377e-c45b-4d96-a785-eac125a03910" containerName="util" Jul 15 08:05:31.488156 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.488140 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a650377e-c45b-4d96-a785-eac125a03910" containerName="extract" Jul 15 08:05:31.488156 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.488145 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a650377e-c45b-4d96-a785-eac125a03910" containerName="extract" Jul 15 08:05:31.488302 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.488189 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a650377e-c45b-4d96-a785-eac125a03910" containerName="extract" Jul 15 08:05:31.492500 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.492480 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.495426 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.495400 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 08:05:31.495558 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.495400 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5958h\"" Jul 15 08:05:31.495558 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.495506 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 08:05:31.495558 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.495529 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 08:05:31.495878 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.495865 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 08:05:31.520003 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.519974 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s"] Jul 15 08:05:31.570051 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.570026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-tmp\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.570148 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.570056 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-manifests\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.570148 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.570073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.570148 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.570094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.570255 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.570187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlrbn\" (UniqueName: \"kubernetes.io/projected/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-kube-api-access-rlrbn\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.648339 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.648319 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:31.671397 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.671370 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.671514 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.671432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlrbn\" (UniqueName: \"kubernetes.io/projected/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-kube-api-access-rlrbn\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.671578 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.671513 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-tmp\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.671578 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.671546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-manifests\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.671578 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.671571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.671938 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.671913 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-manifests\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.673665 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.673644 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-tmp\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.673796 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.673780 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.673900 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.673887 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.686693 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.686676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlrbn\" (UniqueName: \"kubernetes.io/projected/bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10-kube-api-access-rlrbn\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-wmk8s\" (UID: \"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.802917 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.802894 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:31.841266 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.841240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:31.949276 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.949167 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s"] Jul 15 08:05:31.952128 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:05:31.952104 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaf33b4_60e3_47c8_b5e8_2a6f44aa1a10.slice/crio-663c9c25281a8923820da6652e7f350d722ec7d2723fee7517d19a3a5d9cb6cd WatchSource:0}: Error finding container 663c9c25281a8923820da6652e7f350d722ec7d2723fee7517d19a3a5d9cb6cd: Status 404 returned error can't find the container with id 663c9c25281a8923820da6652e7f350d722ec7d2723fee7517d19a3a5d9cb6cd Jul 15 08:05:31.978301 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:31.978277 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:32.070085 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:32.070064 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:32.272797 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:32.272740 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:32.319230 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:32.319205 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" event={"ID":"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10","Type":"ContainerStarted","Data":"663c9c25281a8923820da6652e7f350d722ec7d2723fee7517d19a3a5d9cb6cd"} Jul 15 08:05:32.366428 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:32.366406 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:35.294033 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:35.294011 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-722zt" Jul 15 08:05:35.335819 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:35.335796 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" event={"ID":"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10","Type":"ContainerStarted","Data":"8e7d909d4977fa87e7e4002380af1192d60c5170b5533f246912b1f3744c9a27"} Jul 15 08:05:36.341980 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:36.341950 2567 generic.go:356] "Generic (PLEG): container finished" podID="bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10" containerID="8e7d909d4977fa87e7e4002380af1192d60c5170b5533f246912b1f3744c9a27" exitCode=0 Jul 15 08:05:36.342317 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:36.342022 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" event={"ID":"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10","Type":"ContainerDied","Data":"8e7d909d4977fa87e7e4002380af1192d60c5170b5533f246912b1f3744c9a27"} Jul 15 08:05:37.348279 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:37.348238 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" event={"ID":"bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10","Type":"ContainerStarted","Data":"02e0cefd8c79c4fa5639568ca2d0b62f27b8b467b16cf9eb184adc0dd375a1cf"} Jul 15 08:05:37.348777 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:37.348403 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:37.371846 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:37.371797 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" podStartSLOduration=1.8295802509999999 podStartE2EDuration="6.371777802s" totalImagesPullingTime="3.534876739s" totalInitContainerRuntime="1.007320812s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:05:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:05:37.371243885 +0000 UTC m=+294.887040724" watchObservedRunningTime="2026-07-15 08:05:37.371777802 +0000 UTC m=+294.887574649" Jul 15 08:05:42.393019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:42.392978 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:42.905656 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:42.905617 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:05:42.906985 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:42.906961 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:05:42.910032 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:42.910015 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 08:05:45.408359 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:45.408321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:48.355341 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:48.355310 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-wmk8s" Jul 15 08:05:50.245186 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.245114 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:50.411562 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.411526 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:50.691928 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.691895 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:50.878478 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.878422 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf"] Jul 15 08:05:50.883141 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.883108 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:50.883629 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.883596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:50.886024 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.886004 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qhqw9\"" Jul 15 08:05:50.887006 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.886983 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 08:05:50.887108 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.887048 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 08:05:50.887171 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.887096 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 08:05:50.887321 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.887306 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 08:05:50.899396 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:50.899372 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf"] Jul 15 08:05:51.016249 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.016167 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c011c351-7f03-49f2-a9d5-83fd4958f78d-tmp\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.016249 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.016228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zpk\" (UniqueName: \"kubernetes.io/projected/c011c351-7f03-49f2-a9d5-83fd4958f78d-kube-api-access-n8zpk\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.016421 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.016264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c011c351-7f03-49f2-a9d5-83fd4958f78d-tls-certs\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.117105 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.117066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zpk\" (UniqueName: \"kubernetes.io/projected/c011c351-7f03-49f2-a9d5-83fd4958f78d-kube-api-access-n8zpk\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.117284 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.117114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c011c351-7f03-49f2-a9d5-83fd4958f78d-tls-certs\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.117284 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.117210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c011c351-7f03-49f2-a9d5-83fd4958f78d-tmp\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.119408 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.119381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c011c351-7f03-49f2-a9d5-83fd4958f78d-tmp\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.119620 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.119603 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c011c351-7f03-49f2-a9d5-83fd4958f78d-tls-certs\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.138939 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.138914 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zpk\" (UniqueName: \"kubernetes.io/projected/c011c351-7f03-49f2-a9d5-83fd4958f78d-kube-api-access-n8zpk\") pod \"kube-auth-proxy-657b79bd8d-xltvf\" (UID: \"c011c351-7f03-49f2-a9d5-83fd4958f78d\") " pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.192928 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.192895 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" Jul 15 08:05:51.344761 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.344728 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf"] Jul 15 08:05:51.347814 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:05:51.347788 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc011c351_7f03_49f2_a9d5_83fd4958f78d.slice/crio-8ace9849b3e4c54a33c58a3f1ee1ba4a262b03722f88cf689ab3750072c0b022 WatchSource:0}: Error finding container 8ace9849b3e4c54a33c58a3f1ee1ba4a262b03722f88cf689ab3750072c0b022: Status 404 returned error can't find the container with id 8ace9849b3e4c54a33c58a3f1ee1ba4a262b03722f88cf689ab3750072c0b022 Jul 15 08:05:51.350127 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.350111 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:05:51.410429 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.410397 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" event={"ID":"c011c351-7f03-49f2-a9d5-83fd4958f78d","Type":"ContainerStarted","Data":"8ace9849b3e4c54a33c58a3f1ee1ba4a262b03722f88cf689ab3750072c0b022"} Jul 15 08:05:51.619037 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.618995 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:51.948503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:51.948377 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:52.073350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:52.073310 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:52.654179 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:52.654071 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:52.767544 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:52.767512 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:52.866756 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:52.866716 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:52.981846 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:52.981753 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:53.946607 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:53.946566 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:55.429401 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:55.429372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" event={"ID":"c011c351-7f03-49f2-a9d5-83fd4958f78d","Type":"ContainerStarted","Data":"ae40e49318d25b565a52c619737e1e40d4b7621a4aef5762eff9a111b1a3be3c"} Jul 15 08:05:55.447519 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:55.447463 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-657b79bd8d-xltvf" podStartSLOduration=1.986649224 podStartE2EDuration="5.447423726s" totalImagesPullingTime="3.460774502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:05:55.446253397 +0000 UTC m=+312.962050245" watchObservedRunningTime="2026-07-15 08:05:55.447423726 +0000 UTC m=+312.963220560" Jul 15 08:05:58.083717 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:58.083681 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:58.190325 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:58.190285 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:59.018020 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:59.017982 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:59.218151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:59.218112 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:05:59.365066 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:05:59.365024 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:03.013230 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:03.013202 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:03.114558 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:03.114527 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:03.225709 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:03.225681 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:03.322325 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:03.322262 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:03.477619 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:03.477589 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:26.068586 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:26.068541 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:52.273713 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:52.273679 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:52.358704 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:52.358653 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:53.425455 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:53.425398 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:06:53.491448 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:06:53.491419 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:07.306246 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.306217 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt"] Jul 15 08:07:07.308654 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.308634 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.311252 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.311230 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 08:07:07.312191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.312165 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 08:07:07.312284 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.312258 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rx7jh\"" Jul 15 08:07:07.318555 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.318535 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt"] Jul 15 08:07:07.370221 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.370192 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.370323 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.370224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzp8\" (UniqueName: \"kubernetes.io/projected/ba248e4a-2f01-47fe-b48e-4c241b1480d7-kube-api-access-pgzp8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.370323 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.370268 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.470930 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.470907 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.471017 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.470962 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.471093 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.471070 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzp8\" (UniqueName: \"kubernetes.io/projected/ba248e4a-2f01-47fe-b48e-4c241b1480d7-kube-api-access-pgzp8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.471254 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.471240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.471289 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.471275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.480457 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.480413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzp8\" (UniqueName: \"kubernetes.io/projected/ba248e4a-2f01-47fe-b48e-4c241b1480d7-kube-api-access-pgzp8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.617829 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.617808 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:07.748583 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.748557 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt"] Jul 15 08:07:07.750458 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:07:07.750410 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba248e4a_2f01_47fe_b48e_4c241b1480d7.slice/crio-a266a2ce00ae05aaaa71cdc9f948c7576a0e0bdb6c7426baf44630a6858c6106 WatchSource:0}: Error finding container a266a2ce00ae05aaaa71cdc9f948c7576a0e0bdb6c7426baf44630a6858c6106: Status 404 returned error can't find the container with id a266a2ce00ae05aaaa71cdc9f948c7576a0e0bdb6c7426baf44630a6858c6106 Jul 15 08:07:07.904551 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.904493 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb"] Jul 15 08:07:07.906802 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.906786 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:07.916955 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.916934 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb"] Jul 15 08:07:07.973486 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.973463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:07.973575 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.973502 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhr2n\" (UniqueName: \"kubernetes.io/projected/488a3ba2-7ee4-47dc-9238-352d8cd763bb-kube-api-access-xhr2n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:07.973698 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:07.973570 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:08.074929 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.074908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:08.075017 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.074952 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhr2n\" (UniqueName: \"kubernetes.io/projected/488a3ba2-7ee4-47dc-9238-352d8cd763bb-kube-api-access-xhr2n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:08.075017 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.074979 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:08.075270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.075254 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:08.075326 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.075310 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:08.083910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.083892 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhr2n\" (UniqueName: \"kubernetes.io/projected/488a3ba2-7ee4-47dc-9238-352d8cd763bb-kube-api-access-xhr2n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:08.214855 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.214806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:08.305906 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.305870 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl"] Jul 15 08:07:08.308999 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.308980 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.318619 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.318593 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl"] Jul 15 08:07:08.346799 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.346771 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb"] Jul 15 08:07:08.348598 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:07:08.348576 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488a3ba2_7ee4_47dc_9238_352d8cd763bb.slice/crio-5187b3390ebcf7624e967f0d513d9c45bc28c4f435c8e78e5be08949c927d9f3 WatchSource:0}: Error finding container 5187b3390ebcf7624e967f0d513d9c45bc28c4f435c8e78e5be08949c927d9f3: Status 404 returned error can't find the container with id 5187b3390ebcf7624e967f0d513d9c45bc28c4f435c8e78e5be08949c927d9f3 Jul 15 08:07:08.376394 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.376364 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.376521 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.376397 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdvv\" (UniqueName: \"kubernetes.io/projected/a1150c90-f7bf-430c-bebd-90f012094eda-kube-api-access-nsdvv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.376521 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.376489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.477719 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.477661 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.477816 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.477727 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.477816 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.477759 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdvv\" (UniqueName: \"kubernetes.io/projected/a1150c90-f7bf-430c-bebd-90f012094eda-kube-api-access-nsdvv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.478034 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.478012 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.478090 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.478048 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.486308 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.486289 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdvv\" (UniqueName: \"kubernetes.io/projected/a1150c90-f7bf-430c-bebd-90f012094eda-kube-api-access-nsdvv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.619565 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.619544 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:08.706117 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.706090 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8"] Jul 15 08:07:08.708691 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.708671 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.721359 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.721336 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8"] Jul 15 08:07:08.747257 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.747192 2567 generic.go:356] "Generic (PLEG): container finished" podID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerID="e9a093451926d12623d206a7891837426a8cefe2da34a7347e91c01e82d0db55" exitCode=0 Jul 15 08:07:08.747376 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.747280 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" event={"ID":"488a3ba2-7ee4-47dc-9238-352d8cd763bb","Type":"ContainerDied","Data":"e9a093451926d12623d206a7891837426a8cefe2da34a7347e91c01e82d0db55"} Jul 15 08:07:08.747376 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.747329 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" event={"ID":"488a3ba2-7ee4-47dc-9238-352d8cd763bb","Type":"ContainerStarted","Data":"5187b3390ebcf7624e967f0d513d9c45bc28c4f435c8e78e5be08949c927d9f3"} Jul 15 08:07:08.748846 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.748826 2567 generic.go:356] "Generic (PLEG): container finished" podID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerID="eaa0982896ee5b84ee1c8d280a6bae9b5671a0074683deccdbf8cec1ffab634e" exitCode=0 Jul 15 08:07:08.748937 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.748906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" event={"ID":"ba248e4a-2f01-47fe-b48e-4c241b1480d7","Type":"ContainerDied","Data":"eaa0982896ee5b84ee1c8d280a6bae9b5671a0074683deccdbf8cec1ffab634e"} Jul 15 08:07:08.748980 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.748946 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" event={"ID":"ba248e4a-2f01-47fe-b48e-4c241b1480d7","Type":"ContainerStarted","Data":"a266a2ce00ae05aaaa71cdc9f948c7576a0e0bdb6c7426baf44630a6858c6106"} Jul 15 08:07:08.752643 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.752625 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl"] Jul 15 08:07:08.756267 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:07:08.756247 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1150c90_f7bf_430c_bebd_90f012094eda.slice/crio-dea4b8838b4042fb131d96400de58f755d19b528f9356a236c39444fef1d045a WatchSource:0}: Error finding container dea4b8838b4042fb131d96400de58f755d19b528f9356a236c39444fef1d045a: Status 404 returned error can't find the container with id dea4b8838b4042fb131d96400de58f755d19b528f9356a236c39444fef1d045a Jul 15 08:07:08.780576 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.780556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.780661 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.780585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.780661 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.780611 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5cbt\" (UniqueName: \"kubernetes.io/projected/085f1648-bcc8-4eea-9071-4dd0d7c35d44-kube-api-access-w5cbt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.881962 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.881929 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.882064 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.882025 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.882125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.882066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5cbt\" (UniqueName: \"kubernetes.io/projected/085f1648-bcc8-4eea-9071-4dd0d7c35d44-kube-api-access-w5cbt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.882393 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.882367 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.882513 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.882400 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:08.890557 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:08.890537 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5cbt\" (UniqueName: \"kubernetes.io/projected/085f1648-bcc8-4eea-9071-4dd0d7c35d44-kube-api-access-w5cbt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:09.019541 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.019490 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:09.143128 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.143104 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8"] Jul 15 08:07:09.144675 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:07:09.144637 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085f1648_bcc8_4eea_9071_4dd0d7c35d44.slice/crio-41a9bcd2805d147567455800b9da2d33213a497e5e78f9f38d014b1b21450bb6 WatchSource:0}: Error finding container 41a9bcd2805d147567455800b9da2d33213a497e5e78f9f38d014b1b21450bb6: Status 404 returned error can't find the container with id 41a9bcd2805d147567455800b9da2d33213a497e5e78f9f38d014b1b21450bb6 Jul 15 08:07:09.757147 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.757095 2567 generic.go:356] "Generic (PLEG): container finished" podID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerID="b6e320cce5de17661745e9c1b2e8d2c9a462db1c05604ab56f41392fe27048b0" exitCode=0 Jul 15 08:07:09.757516 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.757181 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" event={"ID":"085f1648-bcc8-4eea-9071-4dd0d7c35d44","Type":"ContainerDied","Data":"b6e320cce5de17661745e9c1b2e8d2c9a462db1c05604ab56f41392fe27048b0"} Jul 15 08:07:09.757516 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.757236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" event={"ID":"085f1648-bcc8-4eea-9071-4dd0d7c35d44","Type":"ContainerStarted","Data":"41a9bcd2805d147567455800b9da2d33213a497e5e78f9f38d014b1b21450bb6"} Jul 15 08:07:09.758855 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.758835 2567 generic.go:356] "Generic (PLEG): container finished" podID="a1150c90-f7bf-430c-bebd-90f012094eda" containerID="cb274dd00b5769a04d457b5e3d1e86e6652698f2264bf5518d55121cf2b9f01a" exitCode=0 Jul 15 08:07:09.758955 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.758922 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" event={"ID":"a1150c90-f7bf-430c-bebd-90f012094eda","Type":"ContainerDied","Data":"cb274dd00b5769a04d457b5e3d1e86e6652698f2264bf5518d55121cf2b9f01a"} Jul 15 08:07:09.759029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.758976 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" event={"ID":"a1150c90-f7bf-430c-bebd-90f012094eda","Type":"ContainerStarted","Data":"dea4b8838b4042fb131d96400de58f755d19b528f9356a236c39444fef1d045a"} Jul 15 08:07:09.761040 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:09.761010 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" event={"ID":"488a3ba2-7ee4-47dc-9238-352d8cd763bb","Type":"ContainerStarted","Data":"0e20082efeb78feff3a047aee809efbd2a2b22cf189c8573c62c01d09b3723ac"} Jul 15 08:07:10.767151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:10.767118 2567 generic.go:356] "Generic (PLEG): container finished" podID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerID="0e20082efeb78feff3a047aee809efbd2a2b22cf189c8573c62c01d09b3723ac" exitCode=0 Jul 15 08:07:10.767562 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:10.767195 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" event={"ID":"488a3ba2-7ee4-47dc-9238-352d8cd763bb","Type":"ContainerDied","Data":"0e20082efeb78feff3a047aee809efbd2a2b22cf189c8573c62c01d09b3723ac"} Jul 15 08:07:10.769059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:10.769036 2567 generic.go:356] "Generic (PLEG): container finished" podID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerID="16f07eb9c3699f0bcca68d4f4feca875ca063e6b9adfd1604e7448494f4865d4" exitCode=0 Jul 15 08:07:10.769157 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:10.769061 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" event={"ID":"ba248e4a-2f01-47fe-b48e-4c241b1480d7","Type":"ContainerDied","Data":"16f07eb9c3699f0bcca68d4f4feca875ca063e6b9adfd1604e7448494f4865d4"} Jul 15 08:07:11.778389 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:11.778361 2567 generic.go:356] "Generic (PLEG): container finished" podID="a1150c90-f7bf-430c-bebd-90f012094eda" containerID="cc2246b184be4b850a055d7bbdbcbdbc096375a8a885378f2f4449e5ad83ec94" exitCode=0 Jul 15 08:07:11.778801 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:11.778460 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" event={"ID":"a1150c90-f7bf-430c-bebd-90f012094eda","Type":"ContainerDied","Data":"cc2246b184be4b850a055d7bbdbcbdbc096375a8a885378f2f4449e5ad83ec94"} Jul 15 08:07:11.780554 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:11.780526 2567 generic.go:356] "Generic (PLEG): container finished" podID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerID="d7ad96f95f8d8f7aff50db8620ddc47152fa60a4007793b20862d057b544d68e" exitCode=0 Jul 15 08:07:11.780656 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:11.780615 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" event={"ID":"488a3ba2-7ee4-47dc-9238-352d8cd763bb","Type":"ContainerDied","Data":"d7ad96f95f8d8f7aff50db8620ddc47152fa60a4007793b20862d057b544d68e"} Jul 15 08:07:11.782765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:11.782742 2567 generic.go:356] "Generic (PLEG): container finished" podID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerID="b5081b555b2f048ed38ca5e46c138cec8ef5a48422134cc1d4a88047ad6fb453" exitCode=0 Jul 15 08:07:11.782881 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:11.782862 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" event={"ID":"ba248e4a-2f01-47fe-b48e-4c241b1480d7","Type":"ContainerDied","Data":"b5081b555b2f048ed38ca5e46c138cec8ef5a48422134cc1d4a88047ad6fb453"} Jul 15 08:07:11.784677 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:11.784658 2567 generic.go:356] "Generic (PLEG): container finished" podID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerID="ee45bcfe725c8d241a1eacaf18f994b9b9d26446989608a351b2f0d086700af9" exitCode=0 Jul 15 08:07:11.784773 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:11.784688 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" event={"ID":"085f1648-bcc8-4eea-9071-4dd0d7c35d44","Type":"ContainerDied","Data":"ee45bcfe725c8d241a1eacaf18f994b9b9d26446989608a351b2f0d086700af9"} Jul 15 08:07:12.791353 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:12.791324 2567 generic.go:356] "Generic (PLEG): container finished" podID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerID="4071bc25e07e9688462a958741c7d257bb95113c1b9254c63e5f1a14504404f7" exitCode=0 Jul 15 08:07:12.791875 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:12.791400 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" event={"ID":"085f1648-bcc8-4eea-9071-4dd0d7c35d44","Type":"ContainerDied","Data":"4071bc25e07e9688462a958741c7d257bb95113c1b9254c63e5f1a14504404f7"} Jul 15 08:07:12.793293 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:12.793271 2567 generic.go:356] "Generic (PLEG): container finished" podID="a1150c90-f7bf-430c-bebd-90f012094eda" containerID="17a9cb0943870b1bf80f1c1bae4a125fb2c4e298cbfe6cfb0c517d4661cb99e4" exitCode=0 Jul 15 08:07:12.793460 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:12.793361 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" event={"ID":"a1150c90-f7bf-430c-bebd-90f012094eda","Type":"ContainerDied","Data":"17a9cb0943870b1bf80f1c1bae4a125fb2c4e298cbfe6cfb0c517d4661cb99e4"} Jul 15 08:07:12.922725 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:12.922692 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:12.948976 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:12.948956 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:13.014877 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.014849 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhr2n\" (UniqueName: \"kubernetes.io/projected/488a3ba2-7ee4-47dc-9238-352d8cd763bb-kube-api-access-xhr2n\") pod \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " Jul 15 08:07:13.015008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.014885 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-util\") pod \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " Jul 15 08:07:13.015008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.014945 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgzp8\" (UniqueName: \"kubernetes.io/projected/ba248e4a-2f01-47fe-b48e-4c241b1480d7-kube-api-access-pgzp8\") pod \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " Jul 15 08:07:13.015008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.014962 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-bundle\") pod \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " Jul 15 08:07:13.015008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.014979 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-bundle\") pod \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\" (UID: \"ba248e4a-2f01-47fe-b48e-4c241b1480d7\") " Jul 15 08:07:13.015008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.015006 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-util\") pod \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\" (UID: \"488a3ba2-7ee4-47dc-9238-352d8cd763bb\") " Jul 15 08:07:13.015513 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.015484 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-bundle" (OuterVolumeSpecName: "bundle") pod "488a3ba2-7ee4-47dc-9238-352d8cd763bb" (UID: "488a3ba2-7ee4-47dc-9238-352d8cd763bb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:13.015627 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.015607 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-bundle" (OuterVolumeSpecName: "bundle") pod "ba248e4a-2f01-47fe-b48e-4c241b1480d7" (UID: "ba248e4a-2f01-47fe-b48e-4c241b1480d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:13.017187 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.017158 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488a3ba2-7ee4-47dc-9238-352d8cd763bb-kube-api-access-xhr2n" (OuterVolumeSpecName: "kube-api-access-xhr2n") pod "488a3ba2-7ee4-47dc-9238-352d8cd763bb" (UID: "488a3ba2-7ee4-47dc-9238-352d8cd763bb"). InnerVolumeSpecName "kube-api-access-xhr2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:07:13.017187 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.017167 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba248e4a-2f01-47fe-b48e-4c241b1480d7-kube-api-access-pgzp8" (OuterVolumeSpecName: "kube-api-access-pgzp8") pod "ba248e4a-2f01-47fe-b48e-4c241b1480d7" (UID: "ba248e4a-2f01-47fe-b48e-4c241b1480d7"). InnerVolumeSpecName "kube-api-access-pgzp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:07:13.022325 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.022294 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-util" (OuterVolumeSpecName: "util") pod "488a3ba2-7ee4-47dc-9238-352d8cd763bb" (UID: "488a3ba2-7ee4-47dc-9238-352d8cd763bb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:13.097373 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.097351 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-util" (OuterVolumeSpecName: "util") pod "ba248e4a-2f01-47fe-b48e-4c241b1480d7" (UID: "ba248e4a-2f01-47fe-b48e-4c241b1480d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:13.116085 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.116066 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgzp8\" (UniqueName: \"kubernetes.io/projected/ba248e4a-2f01-47fe-b48e-4c241b1480d7-kube-api-access-pgzp8\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:13.116085 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.116084 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:13.116203 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.116093 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:13.116203 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.116101 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488a3ba2-7ee4-47dc-9238-352d8cd763bb-util\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:13.116203 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.116109 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhr2n\" (UniqueName: \"kubernetes.io/projected/488a3ba2-7ee4-47dc-9238-352d8cd763bb-kube-api-access-xhr2n\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:13.116203 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.116117 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba248e4a-2f01-47fe-b48e-4c241b1480d7-util\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:13.799521 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.799493 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" Jul 15 08:07:13.799957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.799489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb" event={"ID":"488a3ba2-7ee4-47dc-9238-352d8cd763bb","Type":"ContainerDied","Data":"5187b3390ebcf7624e967f0d513d9c45bc28c4f435c8e78e5be08949c927d9f3"} Jul 15 08:07:13.799957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.799623 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5187b3390ebcf7624e967f0d513d9c45bc28c4f435c8e78e5be08949c927d9f3" Jul 15 08:07:13.801421 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.801391 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" Jul 15 08:07:13.801561 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.801418 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt" event={"ID":"ba248e4a-2f01-47fe-b48e-4c241b1480d7","Type":"ContainerDied","Data":"a266a2ce00ae05aaaa71cdc9f948c7576a0e0bdb6c7426baf44630a6858c6106"} Jul 15 08:07:13.801561 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.801478 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a266a2ce00ae05aaaa71cdc9f948c7576a0e0bdb6c7426baf44630a6858c6106" Jul 15 08:07:13.931730 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.931710 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:13.959414 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:13.959391 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:14.024702 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.024678 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-util\") pod \"a1150c90-f7bf-430c-bebd-90f012094eda\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " Jul 15 08:07:14.024851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.024713 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-bundle\") pod \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " Jul 15 08:07:14.024851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.024759 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsdvv\" (UniqueName: \"kubernetes.io/projected/a1150c90-f7bf-430c-bebd-90f012094eda-kube-api-access-nsdvv\") pod \"a1150c90-f7bf-430c-bebd-90f012094eda\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " Jul 15 08:07:14.024851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.024780 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5cbt\" (UniqueName: \"kubernetes.io/projected/085f1648-bcc8-4eea-9071-4dd0d7c35d44-kube-api-access-w5cbt\") pod \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " Jul 15 08:07:14.024851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.024808 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-bundle\") pod \"a1150c90-f7bf-430c-bebd-90f012094eda\" (UID: \"a1150c90-f7bf-430c-bebd-90f012094eda\") " Jul 15 08:07:14.024851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.024850 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-util\") pod \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\" (UID: \"085f1648-bcc8-4eea-9071-4dd0d7c35d44\") " Jul 15 08:07:14.025361 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.025334 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-bundle" (OuterVolumeSpecName: "bundle") pod "a1150c90-f7bf-430c-bebd-90f012094eda" (UID: "a1150c90-f7bf-430c-bebd-90f012094eda"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:14.025608 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.025582 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-bundle" (OuterVolumeSpecName: "bundle") pod "085f1648-bcc8-4eea-9071-4dd0d7c35d44" (UID: "085f1648-bcc8-4eea-9071-4dd0d7c35d44"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:14.027036 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.027012 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085f1648-bcc8-4eea-9071-4dd0d7c35d44-kube-api-access-w5cbt" (OuterVolumeSpecName: "kube-api-access-w5cbt") pod "085f1648-bcc8-4eea-9071-4dd0d7c35d44" (UID: "085f1648-bcc8-4eea-9071-4dd0d7c35d44"). InnerVolumeSpecName "kube-api-access-w5cbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:07:14.027132 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.027063 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1150c90-f7bf-430c-bebd-90f012094eda-kube-api-access-nsdvv" (OuterVolumeSpecName: "kube-api-access-nsdvv") pod "a1150c90-f7bf-430c-bebd-90f012094eda" (UID: "a1150c90-f7bf-430c-bebd-90f012094eda"). InnerVolumeSpecName "kube-api-access-nsdvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:07:14.030517 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.030498 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-util" (OuterVolumeSpecName: "util") pod "085f1648-bcc8-4eea-9071-4dd0d7c35d44" (UID: "085f1648-bcc8-4eea-9071-4dd0d7c35d44"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:14.125605 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.125583 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-util\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:14.125605 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.125605 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085f1648-bcc8-4eea-9071-4dd0d7c35d44-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:14.125730 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.125619 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdvv\" (UniqueName: \"kubernetes.io/projected/a1150c90-f7bf-430c-bebd-90f012094eda-kube-api-access-nsdvv\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:14.125730 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.125630 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5cbt\" (UniqueName: \"kubernetes.io/projected/085f1648-bcc8-4eea-9071-4dd0d7c35d44-kube-api-access-w5cbt\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:14.125730 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.125638 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:14.133603 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.133582 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-util" (OuterVolumeSpecName: "util") pod "a1150c90-f7bf-430c-bebd-90f012094eda" (UID: "a1150c90-f7bf-430c-bebd-90f012094eda"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:14.226231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.226212 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1150c90-f7bf-430c-bebd-90f012094eda-util\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:07:14.808821 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.808795 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" Jul 15 08:07:14.808821 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.808802 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8" event={"ID":"085f1648-bcc8-4eea-9071-4dd0d7c35d44","Type":"ContainerDied","Data":"41a9bcd2805d147567455800b9da2d33213a497e5e78f9f38d014b1b21450bb6"} Jul 15 08:07:14.809240 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.808859 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41a9bcd2805d147567455800b9da2d33213a497e5e78f9f38d014b1b21450bb6" Jul 15 08:07:14.810983 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.810959 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" Jul 15 08:07:14.811082 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.810979 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl" event={"ID":"a1150c90-f7bf-430c-bebd-90f012094eda","Type":"ContainerDied","Data":"dea4b8838b4042fb131d96400de58f755d19b528f9356a236c39444fef1d045a"} Jul 15 08:07:14.811082 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:14.811017 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea4b8838b4042fb131d96400de58f755d19b528f9356a236c39444fef1d045a" Jul 15 08:07:16.048487 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:16.048454 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:16.102807 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:16.102784 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:20.917530 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:20.917477 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:21.015344 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:21.015314 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:24.102399 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:24.102364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:24.907957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:24.907916 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:27.325682 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:27.325649 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:27.420899 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:27.420851 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:27.530182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:27.530157 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:28.115945 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:28.115906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:28.688234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:28.688204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:28.789504 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:28.789466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:30.545967 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:30.545933 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:31.833914 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:31.833875 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:33.696271 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:33.696240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:33.844071 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:33.844038 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:36.540652 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:36.540623 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:36.647302 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:36.647278 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:41.704475 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:41.704421 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:41.803362 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:41.803332 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:42.214895 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:42.214859 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:42.382148 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:42.382120 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:43.376382 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376353 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-668lz"] Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376643 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerName="extract" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376653 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerName="extract" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376665 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1150c90-f7bf-430c-bebd-90f012094eda" containerName="util" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376671 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1150c90-f7bf-430c-bebd-90f012094eda" containerName="util" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376679 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerName="pull" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376684 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerName="pull" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376692 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerName="pull" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376699 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerName="pull" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376708 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerName="util" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376713 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerName="util" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376720 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerName="extract" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376725 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerName="extract" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376732 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerName="pull" Jul 15 08:07:43.376735 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376737 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerName="pull" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376745 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerName="util" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376750 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerName="util" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376756 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerName="extract" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376761 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerName="extract" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376768 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerName="util" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376773 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerName="util" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376778 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1150c90-f7bf-430c-bebd-90f012094eda" containerName="pull" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376782 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1150c90-f7bf-430c-bebd-90f012094eda" containerName="pull" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376787 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1150c90-f7bf-430c-bebd-90f012094eda" containerName="extract" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376792 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1150c90-f7bf-430c-bebd-90f012094eda" containerName="extract" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376835 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="488a3ba2-7ee4-47dc-9238-352d8cd763bb" containerName="extract" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376842 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1150c90-f7bf-430c-bebd-90f012094eda" containerName="extract" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376849 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="085f1648-bcc8-4eea-9071-4dd0d7c35d44" containerName="extract" Jul 15 08:07:43.377125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.376857 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba248e4a-2f01-47fe-b48e-4c241b1480d7" containerName="extract" Jul 15 08:07:43.379754 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.379737 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.382302 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.382271 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 08:07:43.382302 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.382287 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rx7jh\"" Jul 15 08:07:43.383040 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.383026 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 08:07:43.383119 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.383048 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 08:07:43.383119 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.383083 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 08:07:43.389227 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.389206 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-668lz"] Jul 15 08:07:43.408473 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.408423 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:43.505744 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.505720 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:43.536316 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.536292 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e24e7682-c745-4d46-8d14-858091923416-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.536405 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.536333 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqvs\" (UniqueName: \"kubernetes.io/projected/e24e7682-c745-4d46-8d14-858091923416-kube-api-access-lrqvs\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.536464 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.536412 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e24e7682-c745-4d46-8d14-858091923416-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.637244 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.637184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e24e7682-c745-4d46-8d14-858091923416-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.637244 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.637223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e24e7682-c745-4d46-8d14-858091923416-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.637393 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.637266 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqvs\" (UniqueName: \"kubernetes.io/projected/e24e7682-c745-4d46-8d14-858091923416-kube-api-access-lrqvs\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.637870 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.637849 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e24e7682-c745-4d46-8d14-858091923416-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.639507 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.639488 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e24e7682-c745-4d46-8d14-858091923416-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.645902 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.645873 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqvs\" (UniqueName: \"kubernetes.io/projected/e24e7682-c745-4d46-8d14-858091923416-kube-api-access-lrqvs\") pod \"kuadrant-console-plugin-66b5774985-668lz\" (UID: \"e24e7682-c745-4d46-8d14-858091923416\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.689157 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.689136 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" Jul 15 08:07:43.820672 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.820648 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-668lz"] Jul 15 08:07:43.822570 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:07:43.822540 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode24e7682_c745_4d46_8d14_858091923416.slice/crio-520c89b2d6f66e7c56a0f5c1f7648dab35beaee1adc36b779ad01bf624ba4d07 WatchSource:0}: Error finding container 520c89b2d6f66e7c56a0f5c1f7648dab35beaee1adc36b779ad01bf624ba4d07: Status 404 returned error can't find the container with id 520c89b2d6f66e7c56a0f5c1f7648dab35beaee1adc36b779ad01bf624ba4d07 Jul 15 08:07:43.946369 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:43.946313 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" event={"ID":"e24e7682-c745-4d46-8d14-858091923416","Type":"ContainerStarted","Data":"520c89b2d6f66e7c56a0f5c1f7648dab35beaee1adc36b779ad01bf624ba4d07"} Jul 15 08:07:45.750234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:45.750196 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:45.915626 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:45.915591 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:52.235610 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:52.235571 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:52.334002 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:52.333955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:55.164886 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:55.164845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:07:55.501006 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:07:55.500930 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:00.031957 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:00.031928 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:00.100244 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:00.100213 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:00.223499 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:00.223430 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:00.289027 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:00.288951 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:03.318043 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:03.318007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:03.419205 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:03.419170 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:07.067486 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:07.067456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" event={"ID":"e24e7682-c745-4d46-8d14-858091923416","Type":"ContainerStarted","Data":"7d2e4276dde184b58b2418d210d2d923d71b85ac05a18e3e72e05f5c2df6f3d3"} Jul 15 08:08:07.083860 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:07.083820 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-668lz" podStartSLOduration=0.936541199 podStartE2EDuration="24.083805453s" totalImagesPullingTime="23.147264254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:08:07.082399385 +0000 UTC m=+444.598196227" watchObservedRunningTime="2026-07-15 08:08:07.083805453 +0000 UTC m=+444.599602286" Jul 15 08:08:10.356634 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:10.356600 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:10.467840 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:10.467812 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:27.759918 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:27.759884 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:27.940770 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:27.940733 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:36.200677 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:36.200648 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:36.308786 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:36.308757 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:42.914034 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:42.914001 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:43.018763 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.018734 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:43.080061 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.080027 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:43.226322 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.226264 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:08:43.256896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.256867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:08:43.256896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.256898 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:08:43.257069 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.256976 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:43.259090 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.259070 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 08:08:43.374402 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.374376 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e010c21-46b3-4512-9c6a-d2e1a49d5956-config-file\") pod \"limitador-limitador-54d6d988d5-fq8cd\" (UID: \"6e010c21-46b3-4512-9c6a-d2e1a49d5956\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:43.374530 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.374407 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpwrg\" (UniqueName: \"kubernetes.io/projected/6e010c21-46b3-4512-9c6a-d2e1a49d5956-kube-api-access-rpwrg\") pod \"limitador-limitador-54d6d988d5-fq8cd\" (UID: \"6e010c21-46b3-4512-9c6a-d2e1a49d5956\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:43.475299 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.475275 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e010c21-46b3-4512-9c6a-d2e1a49d5956-config-file\") pod \"limitador-limitador-54d6d988d5-fq8cd\" (UID: \"6e010c21-46b3-4512-9c6a-d2e1a49d5956\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:43.475396 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.475305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwrg\" (UniqueName: \"kubernetes.io/projected/6e010c21-46b3-4512-9c6a-d2e1a49d5956-kube-api-access-rpwrg\") pod \"limitador-limitador-54d6d988d5-fq8cd\" (UID: \"6e010c21-46b3-4512-9c6a-d2e1a49d5956\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:43.475866 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.475850 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e010c21-46b3-4512-9c6a-d2e1a49d5956-config-file\") pod \"limitador-limitador-54d6d988d5-fq8cd\" (UID: \"6e010c21-46b3-4512-9c6a-d2e1a49d5956\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:43.485483 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.485424 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwrg\" (UniqueName: \"kubernetes.io/projected/6e010c21-46b3-4512-9c6a-d2e1a49d5956-kube-api-access-rpwrg\") pod \"limitador-limitador-54d6d988d5-fq8cd\" (UID: \"6e010c21-46b3-4512-9c6a-d2e1a49d5956\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:43.565904 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.565878 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:43.693820 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.693769 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:08:43.695883 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:08:43.695851 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e010c21_46b3_4512_9c6a_d2e1a49d5956.slice/crio-4bd74ea49cf724ce43e8a4277857077a78addd13bd1face8ed97c4e3ef3d642e WatchSource:0}: Error finding container 4bd74ea49cf724ce43e8a4277857077a78addd13bd1face8ed97c4e3ef3d642e: Status 404 returned error can't find the container with id 4bd74ea49cf724ce43e8a4277857077a78addd13bd1face8ed97c4e3ef3d642e Jul 15 08:08:43.717758 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.717732 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-tczj8"] Jul 15 08:08:43.720413 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.720392 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tczj8" Jul 15 08:08:43.723609 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.723589 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grskd\"" Jul 15 08:08:43.731784 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.731760 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tczj8"] Jul 15 08:08:43.751710 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.751656 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:43.826849 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.826824 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:08:43.878137 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.878112 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ngr\" (UniqueName: \"kubernetes.io/projected/a11863b1-0930-4d19-ad8a-cbab69878d7f-kube-api-access-c6ngr\") pod \"authorino-58cd955675-tczj8\" (UID: \"a11863b1-0930-4d19-ad8a-cbab69878d7f\") " pod="kuadrant-system/authorino-58cd955675-tczj8" Jul 15 08:08:43.978639 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.978613 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ngr\" (UniqueName: \"kubernetes.io/projected/a11863b1-0930-4d19-ad8a-cbab69878d7f-kube-api-access-c6ngr\") pod \"authorino-58cd955675-tczj8\" (UID: \"a11863b1-0930-4d19-ad8a-cbab69878d7f\") " pod="kuadrant-system/authorino-58cd955675-tczj8" Jul 15 08:08:43.992092 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:43.992069 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ngr\" (UniqueName: \"kubernetes.io/projected/a11863b1-0930-4d19-ad8a-cbab69878d7f-kube-api-access-c6ngr\") pod \"authorino-58cd955675-tczj8\" (UID: \"a11863b1-0930-4d19-ad8a-cbab69878d7f\") " pod="kuadrant-system/authorino-58cd955675-tczj8" Jul 15 08:08:44.028427 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:44.028376 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tczj8" Jul 15 08:08:44.152803 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:44.152777 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tczj8"] Jul 15 08:08:44.154494 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:08:44.154472 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda11863b1_0930_4d19_ad8a_cbab69878d7f.slice/crio-ba2bea19e4ea2584029495c7186eb50301eb4eb3caf46eff2ab0633f5ebff788 WatchSource:0}: Error finding container ba2bea19e4ea2584029495c7186eb50301eb4eb3caf46eff2ab0633f5ebff788: Status 404 returned error can't find the container with id ba2bea19e4ea2584029495c7186eb50301eb4eb3caf46eff2ab0633f5ebff788 Jul 15 08:08:44.233364 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:44.233331 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tczj8" event={"ID":"a11863b1-0930-4d19-ad8a-cbab69878d7f","Type":"ContainerStarted","Data":"ba2bea19e4ea2584029495c7186eb50301eb4eb3caf46eff2ab0633f5ebff788"} Jul 15 08:08:44.234812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:44.234789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" event={"ID":"6e010c21-46b3-4512-9c6a-d2e1a49d5956","Type":"ContainerStarted","Data":"4bd74ea49cf724ce43e8a4277857077a78addd13bd1face8ed97c4e3ef3d642e"} Jul 15 08:08:48.265306 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:48.265210 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" event={"ID":"6e010c21-46b3-4512-9c6a-d2e1a49d5956","Type":"ContainerStarted","Data":"9931ccbc9e9dea085b0c54813d73c26cbed9641778117b97d1994457d13eb9a8"} Jul 15 08:08:48.265306 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:48.265277 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:08:48.266689 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:48.266657 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tczj8" event={"ID":"a11863b1-0930-4d19-ad8a-cbab69878d7f","Type":"ContainerStarted","Data":"f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371"} Jul 15 08:08:48.284321 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:48.284277 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" podStartSLOduration=1.070392959 podStartE2EDuration="5.284262975s" totalImagesPullingTime="4.213870016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:08:48.28258746 +0000 UTC m=+485.798384294" watchObservedRunningTime="2026-07-15 08:08:48.284262975 +0000 UTC m=+485.800059809" Jul 15 08:08:48.304168 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:48.304129 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-tczj8" podStartSLOduration=1.508152231 podStartE2EDuration="5.304116773s" totalImagesPullingTime="3.795964542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:08:48.301954136 +0000 UTC m=+485.817750969" watchObservedRunningTime="2026-07-15 08:08:48.304116773 +0000 UTC m=+485.819913607" Jul 15 08:08:48.529534 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:48.529466 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tczj8"] Jul 15 08:08:50.275673 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:50.275636 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-tczj8" podUID="a11863b1-0930-4d19-ad8a-cbab69878d7f" containerName="authorino" containerID="cri-o://f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371" gracePeriod=30 Jul 15 08:08:50.511208 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:50.511186 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tczj8" Jul 15 08:08:50.633244 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:50.633217 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6ngr\" (UniqueName: \"kubernetes.io/projected/a11863b1-0930-4d19-ad8a-cbab69878d7f-kube-api-access-c6ngr\") pod \"a11863b1-0930-4d19-ad8a-cbab69878d7f\" (UID: \"a11863b1-0930-4d19-ad8a-cbab69878d7f\") " Jul 15 08:08:50.635219 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:50.635191 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11863b1-0930-4d19-ad8a-cbab69878d7f-kube-api-access-c6ngr" (OuterVolumeSpecName: "kube-api-access-c6ngr") pod "a11863b1-0930-4d19-ad8a-cbab69878d7f" (UID: "a11863b1-0930-4d19-ad8a-cbab69878d7f"). InnerVolumeSpecName "kube-api-access-c6ngr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:08:50.734298 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:50.734270 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6ngr\" (UniqueName: \"kubernetes.io/projected/a11863b1-0930-4d19-ad8a-cbab69878d7f-kube-api-access-c6ngr\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:08:51.281287 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.281252 2567 generic.go:356] "Generic (PLEG): container finished" podID="a11863b1-0930-4d19-ad8a-cbab69878d7f" containerID="f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371" exitCode=0 Jul 15 08:08:51.281726 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.281307 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tczj8" Jul 15 08:08:51.281726 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.281329 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tczj8" event={"ID":"a11863b1-0930-4d19-ad8a-cbab69878d7f","Type":"ContainerDied","Data":"f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371"} Jul 15 08:08:51.281726 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.281372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tczj8" event={"ID":"a11863b1-0930-4d19-ad8a-cbab69878d7f","Type":"ContainerDied","Data":"ba2bea19e4ea2584029495c7186eb50301eb4eb3caf46eff2ab0633f5ebff788"} Jul 15 08:08:51.281726 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.281388 2567 scope.go:117] "RemoveContainer" containerID="f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371" Jul 15 08:08:51.289146 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.289130 2567 scope.go:117] "RemoveContainer" containerID="f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371" Jul 15 08:08:51.289383 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:08:51.289364 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371\": container with ID starting with f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371 not found: ID does not exist" containerID="f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371" Jul 15 08:08:51.289422 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.289391 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371"} err="failed to get container status \"f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371\": rpc error: code = NotFound desc = could not find container \"f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371\": container with ID starting with f2dae2b9be832991b117bc32f62926b65dea4597faa2184042a1079b8b02a371 not found: ID does not exist" Jul 15 08:08:51.307474 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.307415 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tczj8"] Jul 15 08:08:51.316005 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:51.315986 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-tczj8"] Jul 15 08:08:53.033366 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:53.033325 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11863b1-0930-4d19-ad8a-cbab69878d7f" path="/var/lib/kubelet/pods/a11863b1-0930-4d19-ad8a-cbab69878d7f/volumes" Jul 15 08:08:59.272122 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:08:59.272094 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fq8cd" Jul 15 08:09:01.277607 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:01.277572 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:01.625914 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:01.625880 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:01.839109 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:01.839080 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:13.962795 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:13.962760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:14.179561 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.179534 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:14.218842 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.218772 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mtwhv"] Jul 15 08:09:14.219099 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.219084 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11863b1-0930-4d19-ad8a-cbab69878d7f" containerName="authorino" Jul 15 08:09:14.219099 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.219100 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11863b1-0930-4d19-ad8a-cbab69878d7f" containerName="authorino" Jul 15 08:09:14.219209 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.219182 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11863b1-0930-4d19-ad8a-cbab69878d7f" containerName="authorino" Jul 15 08:09:14.221015 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.221000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mtwhv" Jul 15 08:09:14.223391 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.223375 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grskd\"" Jul 15 08:09:14.234465 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.234427 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mtwhv"] Jul 15 08:09:14.407052 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.407027 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxnnf\" (UniqueName: \"kubernetes.io/projected/6fb0a482-0f87-4a99-9822-09326f4bb58e-kube-api-access-fxnnf\") pod \"authorino-5d4b67c446-mtwhv\" (UID: \"6fb0a482-0f87-4a99-9822-09326f4bb58e\") " pod="kuadrant-system/authorino-5d4b67c446-mtwhv" Jul 15 08:09:14.504810 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.504745 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mtwhv"] Jul 15 08:09:14.504964 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:09:14.504945 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fxnnf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-mtwhv" podUID="6fb0a482-0f87-4a99-9822-09326f4bb58e" Jul 15 08:09:14.507948 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.507927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxnnf\" (UniqueName: \"kubernetes.io/projected/6fb0a482-0f87-4a99-9822-09326f4bb58e-kube-api-access-fxnnf\") pod \"authorino-5d4b67c446-mtwhv\" (UID: \"6fb0a482-0f87-4a99-9822-09326f4bb58e\") " pod="kuadrant-system/authorino-5d4b67c446-mtwhv" Jul 15 08:09:14.520170 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.520145 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxnnf\" (UniqueName: \"kubernetes.io/projected/6fb0a482-0f87-4a99-9822-09326f4bb58e-kube-api-access-fxnnf\") pod \"authorino-5d4b67c446-mtwhv\" (UID: \"6fb0a482-0f87-4a99-9822-09326f4bb58e\") " pod="kuadrant-system/authorino-5d4b67c446-mtwhv" Jul 15 08:09:14.563874 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.563851 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-594d55d7bb-d98w2"] Jul 15 08:09:14.567060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.567045 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:09:14.570003 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.569985 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 08:09:14.577058 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.577036 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-594d55d7bb-d98w2"] Jul 15 08:09:14.709052 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.709026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6142fb3c-92f4-4e83-8896-b1eec341913b-tls-cert\") pod \"authorino-594d55d7bb-d98w2\" (UID: \"6142fb3c-92f4-4e83-8896-b1eec341913b\") " pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:09:14.709145 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.709084 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxlfj\" (UniqueName: \"kubernetes.io/projected/6142fb3c-92f4-4e83-8896-b1eec341913b-kube-api-access-vxlfj\") pod \"authorino-594d55d7bb-d98w2\" (UID: \"6142fb3c-92f4-4e83-8896-b1eec341913b\") " pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:09:14.810323 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.810266 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6142fb3c-92f4-4e83-8896-b1eec341913b-tls-cert\") pod \"authorino-594d55d7bb-d98w2\" (UID: \"6142fb3c-92f4-4e83-8896-b1eec341913b\") " pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:09:14.810401 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.810329 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxlfj\" (UniqueName: \"kubernetes.io/projected/6142fb3c-92f4-4e83-8896-b1eec341913b-kube-api-access-vxlfj\") pod \"authorino-594d55d7bb-d98w2\" (UID: \"6142fb3c-92f4-4e83-8896-b1eec341913b\") " pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:09:14.812714 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.812694 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6142fb3c-92f4-4e83-8896-b1eec341913b-tls-cert\") pod \"authorino-594d55d7bb-d98w2\" (UID: \"6142fb3c-92f4-4e83-8896-b1eec341913b\") " pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:09:14.819082 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.819060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxlfj\" (UniqueName: \"kubernetes.io/projected/6142fb3c-92f4-4e83-8896-b1eec341913b-kube-api-access-vxlfj\") pod \"authorino-594d55d7bb-d98w2\" (UID: \"6142fb3c-92f4-4e83-8896-b1eec341913b\") " pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:09:14.876029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:14.876010 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:09:15.016660 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:15.016637 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-594d55d7bb-d98w2"] Jul 15 08:09:15.018736 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:09:15.018704 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6142fb3c_92f4_4e83_8896_b1eec341913b.slice/crio-c8712db43866ee5bc76e2b536ca7c22a696ec3ada62436b08327611207c1bb9c WatchSource:0}: Error finding container c8712db43866ee5bc76e2b536ca7c22a696ec3ada62436b08327611207c1bb9c: Status 404 returned error can't find the container with id c8712db43866ee5bc76e2b536ca7c22a696ec3ada62436b08327611207c1bb9c Jul 15 08:09:15.392009 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:15.391982 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-594d55d7bb-d98w2" event={"ID":"6142fb3c-92f4-4e83-8896-b1eec341913b","Type":"ContainerStarted","Data":"c8712db43866ee5bc76e2b536ca7c22a696ec3ada62436b08327611207c1bb9c"} Jul 15 08:09:15.392117 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:15.392010 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mtwhv" Jul 15 08:09:15.397185 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:15.397168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mtwhv" Jul 15 08:09:15.516217 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:15.516185 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxnnf\" (UniqueName: \"kubernetes.io/projected/6fb0a482-0f87-4a99-9822-09326f4bb58e-kube-api-access-fxnnf\") pod \"6fb0a482-0f87-4a99-9822-09326f4bb58e\" (UID: \"6fb0a482-0f87-4a99-9822-09326f4bb58e\") " Jul 15 08:09:15.518228 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:15.518201 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb0a482-0f87-4a99-9822-09326f4bb58e-kube-api-access-fxnnf" (OuterVolumeSpecName: "kube-api-access-fxnnf") pod "6fb0a482-0f87-4a99-9822-09326f4bb58e" (UID: "6fb0a482-0f87-4a99-9822-09326f4bb58e"). InnerVolumeSpecName "kube-api-access-fxnnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:09:15.617180 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:15.617149 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxnnf\" (UniqueName: \"kubernetes.io/projected/6fb0a482-0f87-4a99-9822-09326f4bb58e-kube-api-access-fxnnf\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:09:16.398559 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:16.398522 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-594d55d7bb-d98w2" event={"ID":"6142fb3c-92f4-4e83-8896-b1eec341913b","Type":"ContainerStarted","Data":"cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe"} Jul 15 08:09:16.398968 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:16.398596 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mtwhv" Jul 15 08:09:16.423717 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:16.423673 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-594d55d7bb-d98w2" podStartSLOduration=2.10607697 podStartE2EDuration="2.423658479s" totalImagesPullingTime="317.581509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:09:16.419846965 +0000 UTC m=+513.935643799" watchObservedRunningTime="2026-07-15 08:09:16.423658479 +0000 UTC m=+513.939455317" Jul 15 08:09:16.499074 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:16.499049 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mtwhv"] Jul 15 08:09:16.503307 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:16.503282 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mtwhv"] Jul 15 08:09:17.033264 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:17.033220 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb0a482-0f87-4a99-9822-09326f4bb58e" path="/var/lib/kubelet/pods/6fb0a482-0f87-4a99-9822-09326f4bb58e/volumes" Jul 15 08:09:21.604509 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:21.604474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:21.705378 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:21.705353 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:25.419606 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:25.419572 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:25.557724 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:25.557697 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:57.009603 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:57.009569 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:57.105820 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:57.105790 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:57.623550 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:57.623521 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:58.047338 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:58.047264 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:58.147524 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:58.147494 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:58.229272 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:58.229243 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:09:58.292747 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:09:58.292720 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:10:08.575181 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:10:08.575148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:10:08.674226 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:10:08.674180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:10:42.936853 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:10:42.936828 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:10:42.937395 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:10:42.937084 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:10:45.356288 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:10:45.356258 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:10:45.927389 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:10:45.927350 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:11:04.464491 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:04.464457 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:11:05.225732 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:05.225647 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:11:08.434450 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:08.434408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:11:09.625415 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:09.625377 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:11:14.557267 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:14.557239 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:11:16.319557 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:16.319528 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:11:17.850685 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:17.850649 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:11:23.226833 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.226799 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758"] Jul 15 08:11:23.229557 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.229534 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.232792 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.232762 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 08:11:23.232792 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.232764 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-schk4\"" Jul 15 08:11:23.232935 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.232798 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 08:11:23.232935 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.232858 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 08:11:23.247722 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.247700 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758"] Jul 15 08:11:23.336873 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.336851 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.336975 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.336896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.336975 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.336928 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b52868-5c08-4866-8d73-81c9ad2b70bf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.336975 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.336955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk557\" (UniqueName: \"kubernetes.io/projected/40b52868-5c08-4866-8d73-81c9ad2b70bf-kube-api-access-wk557\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.337077 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.336979 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.337077 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.337046 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.405307 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.405284 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:11:23.437353 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437329 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.437433 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437366 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.437433 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b52868-5c08-4866-8d73-81c9ad2b70bf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.437433 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437425 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk557\" (UniqueName: \"kubernetes.io/projected/40b52868-5c08-4866-8d73-81c9ad2b70bf-kube-api-access-wk557\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.437575 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437485 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.437575 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437520 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.437813 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437790 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.437881 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437831 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.437932 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.437884 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.439712 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.439691 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b52868-5c08-4866-8d73-81c9ad2b70bf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.439832 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.439812 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b52868-5c08-4866-8d73-81c9ad2b70bf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.446908 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.446884 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk557\" (UniqueName: \"kubernetes.io/projected/40b52868-5c08-4866-8d73-81c9ad2b70bf-kube-api-access-wk557\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rs758\" (UID: \"40b52868-5c08-4866-8d73-81c9ad2b70bf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.539173 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.539117 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:23.670838 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.670814 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758"] Jul 15 08:11:23.672769 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:11:23.672742 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b52868_5c08_4866_8d73_81c9ad2b70bf.slice/crio-aaf50a59f0014b45148522abc640fb31e26946b47bf73badd74d42595b55a6b4 WatchSource:0}: Error finding container aaf50a59f0014b45148522abc640fb31e26946b47bf73badd74d42595b55a6b4: Status 404 returned error can't find the container with id aaf50a59f0014b45148522abc640fb31e26946b47bf73badd74d42595b55a6b4 Jul 15 08:11:23.674050 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:23.674034 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:11:24.008991 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:24.008944 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" event={"ID":"40b52868-5c08-4866-8d73-81c9ad2b70bf","Type":"ContainerStarted","Data":"aaf50a59f0014b45148522abc640fb31e26946b47bf73badd74d42595b55a6b4"} Jul 15 08:11:25.015480 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:25.015383 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" event={"ID":"40b52868-5c08-4866-8d73-81c9ad2b70bf","Type":"ContainerStarted","Data":"5717c4c57cd850a966ccead5d0e37fcaab9708311c6b33ec9543bf7c3d500a02"} Jul 15 08:11:25.035349 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:25.035306 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" podStartSLOduration=0.974037335 podStartE2EDuration="2.035291447s" totalImagesPullingTime="1.061254112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:11:25.033274787 +0000 UTC m=+642.549071631" watchObservedRunningTime="2026-07-15 08:11:25.035291447 +0000 UTC m=+642.551088281" Jul 15 08:11:25.426852 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:25.426818 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:11:28.820672 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.820638 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k"] Jul 15 08:11:28.823203 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.823178 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.825906 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.825885 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 08:11:28.833567 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.833545 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k"] Jul 15 08:11:28.856690 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.856665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:11:28.880749 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.880724 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.880822 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.880756 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2ws\" (UniqueName: \"kubernetes.io/projected/c4a43e78-af11-445c-83aa-c3dc3efc0e17-kube-api-access-7k2ws\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.880822 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.880776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.880891 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.880847 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.880891 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.880871 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.880954 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.880892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a43e78-af11-445c-83aa-c3dc3efc0e17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981338 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2ws\" (UniqueName: \"kubernetes.io/projected/c4a43e78-af11-445c-83aa-c3dc3efc0e17-kube-api-access-7k2ws\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981424 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981424 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981424 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981569 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a43e78-af11-445c-83aa-c3dc3efc0e17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981569 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981768 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981828 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981777 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.981880 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.981830 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.983633 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.983610 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a43e78-af11-445c-83aa-c3dc3efc0e17-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.983908 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.983892 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a43e78-af11-445c-83aa-c3dc3efc0e17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:28.989580 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:28.989560 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2ws\" (UniqueName: \"kubernetes.io/projected/c4a43e78-af11-445c-83aa-c3dc3efc0e17-kube-api-access-7k2ws\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k\" (UID: \"c4a43e78-af11-445c-83aa-c3dc3efc0e17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:29.134021 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:29.133996 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:29.261450 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:29.261408 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k"] Jul 15 08:11:29.263069 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:11:29.263043 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4a43e78_af11_445c_83aa_c3dc3efc0e17.slice/crio-f54640211da909a45092f279c5b8a414df329f8305587ecdea06c39a03aff3f2 WatchSource:0}: Error finding container f54640211da909a45092f279c5b8a414df329f8305587ecdea06c39a03aff3f2: Status 404 returned error can't find the container with id f54640211da909a45092f279c5b8a414df329f8305587ecdea06c39a03aff3f2 Jul 15 08:11:30.041456 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:30.041351 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" event={"ID":"c4a43e78-af11-445c-83aa-c3dc3efc0e17","Type":"ContainerStarted","Data":"dad88d1cb91b03dfcece078c48aa7097a2ae9ca16992977d6b2db7d81adee891"} Jul 15 08:11:30.041456 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:30.041390 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" event={"ID":"c4a43e78-af11-445c-83aa-c3dc3efc0e17","Type":"ContainerStarted","Data":"f54640211da909a45092f279c5b8a414df329f8305587ecdea06c39a03aff3f2"} Jul 15 08:11:30.059160 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:30.059116 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" podStartSLOduration=1.7269128249999999 podStartE2EDuration="2.059101299s" totalImagesPullingTime="332.188474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:11:30.057731104 +0000 UTC m=+647.573527941" watchObservedRunningTime="2026-07-15 08:11:30.059101299 +0000 UTC m=+647.574898133" Jul 15 08:11:31.081753 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:31.081716 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:11:33.539677 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:33.539641 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:33.540077 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:33.539699 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:33.552305 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:33.552276 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:34.071476 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:34.071425 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rs758" Jul 15 08:11:37.127359 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:37.127326 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:11:39.135062 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:39.135027 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:39.135062 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:39.135069 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:39.147264 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:39.147237 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:40.106002 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:40.105976 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k" Jul 15 08:11:56.675137 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:56.675096 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:11:56.731697 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:11:56.731668 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:12:29.682607 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:29.682566 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:12:40.277385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:40.277346 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:12:40.401755 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:40.401723 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:12:41.562323 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:41.562281 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:12:41.663446 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:41.663399 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-594d55d7bb-d98w2"] Jul 15 08:12:41.663676 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:41.663637 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-594d55d7bb-d98w2" podUID="6142fb3c-92f4-4e83-8896-b1eec341913b" containerName="authorino" containerID="cri-o://cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe" gracePeriod=30 Jul 15 08:12:41.710943 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:41.710915 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:12:41.907094 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:41.907069 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:12:42.037407 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.037370 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6142fb3c-92f4-4e83-8896-b1eec341913b-tls-cert\") pod \"6142fb3c-92f4-4e83-8896-b1eec341913b\" (UID: \"6142fb3c-92f4-4e83-8896-b1eec341913b\") " Jul 15 08:12:42.037562 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.037417 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxlfj\" (UniqueName: \"kubernetes.io/projected/6142fb3c-92f4-4e83-8896-b1eec341913b-kube-api-access-vxlfj\") pod \"6142fb3c-92f4-4e83-8896-b1eec341913b\" (UID: \"6142fb3c-92f4-4e83-8896-b1eec341913b\") " Jul 15 08:12:42.039568 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.039540 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6142fb3c-92f4-4e83-8896-b1eec341913b-kube-api-access-vxlfj" (OuterVolumeSpecName: "kube-api-access-vxlfj") pod "6142fb3c-92f4-4e83-8896-b1eec341913b" (UID: "6142fb3c-92f4-4e83-8896-b1eec341913b"). InnerVolumeSpecName "kube-api-access-vxlfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:12:42.047487 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.047432 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6142fb3c-92f4-4e83-8896-b1eec341913b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6142fb3c-92f4-4e83-8896-b1eec341913b" (UID: "6142fb3c-92f4-4e83-8896-b1eec341913b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:12:42.139007 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.138986 2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6142fb3c-92f4-4e83-8896-b1eec341913b-tls-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:12:42.139093 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.139007 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxlfj\" (UniqueName: \"kubernetes.io/projected/6142fb3c-92f4-4e83-8896-b1eec341913b-kube-api-access-vxlfj\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:12:42.392546 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.392480 2567 generic.go:356] "Generic (PLEG): container finished" podID="6142fb3c-92f4-4e83-8896-b1eec341913b" containerID="cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe" exitCode=0 Jul 15 08:12:42.392546 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.392517 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-594d55d7bb-d98w2" Jul 15 08:12:42.392679 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.392546 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-594d55d7bb-d98w2" event={"ID":"6142fb3c-92f4-4e83-8896-b1eec341913b","Type":"ContainerDied","Data":"cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe"} Jul 15 08:12:42.392679 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.392578 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-594d55d7bb-d98w2" event={"ID":"6142fb3c-92f4-4e83-8896-b1eec341913b","Type":"ContainerDied","Data":"c8712db43866ee5bc76e2b536ca7c22a696ec3ada62436b08327611207c1bb9c"} Jul 15 08:12:42.392679 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.392594 2567 scope.go:117] "RemoveContainer" containerID="cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe" Jul 15 08:12:42.401180 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.401164 2567 scope.go:117] "RemoveContainer" containerID="cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe" Jul 15 08:12:42.401425 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:12:42.401409 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe\": container with ID starting with cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe not found: ID does not exist" containerID="cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe" Jul 15 08:12:42.401514 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.401454 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe"} err="failed to get container status \"cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe\": rpc error: code = NotFound desc = could not find container \"cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe\": container with ID starting with cfe211faf5a3516b6bd4a02883c4876bf6162de2739b533746bcdefb433419fe not found: ID does not exist" Jul 15 08:12:42.426279 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.426251 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-594d55d7bb-d98w2"] Jul 15 08:12:42.436420 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:42.436399 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-594d55d7bb-d98w2"] Jul 15 08:12:43.033042 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:43.033013 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6142fb3c-92f4-4e83-8896-b1eec341913b" path="/var/lib/kubelet/pods/6142fb3c-92f4-4e83-8896-b1eec341913b/volumes" Jul 15 08:12:50.335092 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:50.335053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:12:53.530219 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:53.530167 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:12:53.613348 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:12:53.613312 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:13:07.327343 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:07.327305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:13:11.579163 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:11.579120 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:13:19.647645 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:19.647611 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:13:27.727333 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:27.727297 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:13:37.929155 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:37.929115 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:13:49.534978 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:49.533801 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:13:55.181981 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:55.181947 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:13:56.704640 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:56.704596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:13:57.731638 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:13:57.731605 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:14:08.026053 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:08.025968 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:19.230386 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:19.230349 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:14:30.928515 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:30.928483 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:14:36.498786 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:36.498748 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:36.597392 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:36.597359 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:36.716085 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:36.716035 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:42.898429 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:42.898394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:42.965470 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:42.965423 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:43.070823 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:43.070793 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:43.268863 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:43.268784 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:49.530338 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:49.530297 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:14:49.596278 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:14:49.596247 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:15:06.844549 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:15:06.844510 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:15:16.035921 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:15:16.035888 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:15:24.839587 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:15:24.839552 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:15:37.942658 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:15:37.942580 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:15:42.969090 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:15:42.969067 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:15:42.969523 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:15:42.969115 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:15:52.611635 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:15:52.611601 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:15:53.974275 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:15:53.974240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:16:05.338552 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:16:05.338516 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:16:15.536579 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:16:15.536543 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:16:40.625847 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:16:40.625817 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:16:52.137278 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:16:52.137248 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:17:14.930925 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:17:14.930850 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:17:25.924864 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:17:25.924832 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:17:41.432606 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:17:41.432567 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:17:53.339333 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:17:53.339297 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:18:08.044469 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:18:08.044424 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:19:47.790748 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:19:47.790713 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:19:47.874424 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:19:47.874379 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:19:51.834271 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:19:51.834234 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:20:07.132720 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:07.132645 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:20:07.223086 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:07.223055 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:20:08.635455 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:08.635414 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:20:17.532056 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:17.532013 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:20:21.555845 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:21.555807 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:20:30.488508 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:30.488472 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:20:42.999916 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:42.999888 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:20:43.001555 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:43.001530 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:20:44.541963 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:44.541929 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:20:54.417408 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:20:54.417371 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:21:04.025992 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:21:04.025959 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:21:14.134922 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:21:14.134885 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:21:23.533577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:21:23.533546 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:21:35.933048 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:21:35.932968 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:21:43.835507 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:21:43.835473 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:21:55.023671 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:21:55.023642 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:22:16.537286 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:22:16.537251 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:22:27.735297 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:22:27.735267 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:22:37.522714 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:22:37.522677 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:22:46.740726 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:22:46.740691 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:22:53.435621 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:22:53.435586 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:23:09.632277 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:23:09.632196 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:23:14.955001 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:23:14.954962 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:23:19.140196 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:23:19.140162 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:23:28.624019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:23:28.623976 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:23:35.941894 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:23:35.941863 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:23:47.183984 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:23:47.183953 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:23:56.831448 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:23:56.831397 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:24:10.033769 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:24:10.033730 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:24:19.442628 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:24:19.442596 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:24:29.026988 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:24:29.026944 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:24:39.931874 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:24:39.931797 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:24:52.031526 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:24:52.031486 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:25:01.940863 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:01.940831 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:25:14.231114 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:14.231079 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:25:26.647472 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:26.647421 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:25:36.144068 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:36.144031 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:25:42.577678 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:42.577646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:42.656561 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:42.656532 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:43.040916 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:43.040891 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:25:43.042939 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:43.042919 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:25:49.347848 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:49.347805 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:49.451333 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:49.451298 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:49.609938 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:49.609867 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:49.704243 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:49.704213 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:51.428454 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:51.428400 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:51.697838 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:51.697756 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:52.058138 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:52.058061 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:52.275811 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:52.275779 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:53.218998 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:53.218969 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:25:59.143413 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:25:59.143377 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:26:36.832297 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:26:36.832260 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:26:36.893877 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:26:36.893849 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:26:55.077074 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:26:55.077030 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:26:55.144433 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:26:55.144405 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:06.374768 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:06.374728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:09.146929 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:09.146895 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:27:11.010563 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:11.010536 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:17.433766 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:17.433729 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:17.540874 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:17.540841 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:21.260586 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:21.260558 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:21.636954 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:21.636923 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:21.718537 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:21.718510 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:21.838621 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:21.838580 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:23.075255 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:23.075225 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:23.189051 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:23.189019 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:23.768490 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:23.768465 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:23.871810 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:23.871782 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:31.628717 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:31.628684 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:27:33.512522 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:33.512494 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:33.572216 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:33.572188 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:27:37.422517 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:27:37.422483 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:27.799498 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:27.799461 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:29.408087 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:29.408053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:30.762338 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:30.762311 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:28:33.547709 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:33.547675 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:34.204564 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:34.204539 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:34.943901 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:34.943870 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:41.018669 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:41.018640 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:41.109153 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:41.109125 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:41.322762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:41.322736 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:41.418230 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:41.418198 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:42.531941 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:42.531915 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:42.625529 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:42.625494 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:42.730988 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:42.730954 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:43.076295 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:43.076264 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:43.307178 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:43.307148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:43.428920 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:43.428895 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:45.173189 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:45.173160 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:28:45.267734 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:28:45.267705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:29:18.827946 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:29:18.827860 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:29:18.927857 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:29:18.927823 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:29:48.276458 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:29:48.276397 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:29:48.377195 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:29:48.377164 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:29:58.255756 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:29:58.255714 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:01.124896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:01.124861 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:30:02.872720 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:02.872664 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:08.967201 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:08.967166 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:09.067558 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:09.067531 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:13.862115 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:13.862084 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:14.161315 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:14.161281 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:15.278450 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:15.278398 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:15.426415 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:15.426386 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:15.622259 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:15.622230 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:15.718047 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:15.718022 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:16.059701 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:16.059623 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:16.159937 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:16.159906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:18.143147 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:18.143108 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:30:25.454940 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:25.454911 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:25.556371 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:25.556339 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:28.873365 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:28.873335 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:30:43.072153 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:43.072125 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:30:43.074049 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:30:43.074028 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:31:30.215510 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:30.215480 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:33.147299 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:33.147266 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:31:34.875706 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:34.875661 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:39.960466 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:39.960419 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:40.948994 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:40.948961 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:41.049572 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:41.049544 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:45.033397 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:45.033359 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:45.643580 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:45.643545 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:45.733407 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:45.733364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:45.836512 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:45.836483 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:47.436110 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:47.436078 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:51.722772 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:51.722730 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:54.883590 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:54.883553 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:54.984123 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:54.984094 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:55.084405 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:55.084373 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:55.190851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:55.190772 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:31:55.673958 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:31:55.673930 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:11.105736 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:11.105696 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:11.206460 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:11.206399 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:21.675987 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:21.675950 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:27.896258 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:27.896222 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:28.006838 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:28.006807 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:28.096889 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:28.096856 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:44.023370 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:44.023328 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:50.838809 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:50.838765 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:50.944261 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:50.944228 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:32:51.038027 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:32:51.037998 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:01.624308 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:01.624268 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:02.485908 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:02.485875 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:08.693282 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:08.693243 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:08.793150 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:08.793114 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:08.932040 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:08.932012 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:09.029866 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:09.029795 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:09.901241 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:09.901206 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:10.016027 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:10.016000 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:10.127407 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:10.127378 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:11.437203 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:11.437169 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:33:11.852111 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:11.852080 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:17.382043 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:17.382007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:18.794005 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:18.793973 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:18.858557 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:18.858532 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:19.153288 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:19.153258 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:33:19.548233 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:19.548157 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:25.997231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:25.997195 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:26.132614 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:26.132581 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:26.215918 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:26.215893 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:26.536658 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:26.536623 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:26.727599 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:26.727563 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:33:26.947131 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:26.947099 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:29.805462 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:29.805402 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:31.803554 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:31.803508 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:34.427828 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:34.427788 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:34.882150 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:34.882084 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:33:35.130654 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:35.130624 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:35.335118 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:35.335035 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:35.641694 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:35.641663 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:42.198920 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:42.198888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:42.292801 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:42.292751 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:53.300846 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:53.300811 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:33:56.141814 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:56.141780 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:33:57.696105 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:33:57.696071 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:03.757362 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:03.757331 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:03.858179 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:03.858152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:08.484015 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:08.483980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:08.733964 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:08.733935 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:08.879761 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:08.879728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:08.983300 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:08.983274 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:09.122402 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:09.122354 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:16.363056 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:16.362998 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:16.576389 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:16.576356 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:16.696393 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:16.696304 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:16.951980 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:16.951889 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:19.863533 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:19.863466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:19.978191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:19.978151 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:23.867834 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:23.867802 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:23.977151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:23.977122 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:30.959457 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:30.959376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:31.304149 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:31.304057 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:31.516074 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:31.516040 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:34.137377 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:34.137326 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:34.240387 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:34.240349 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:34.339018 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:34.338987 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:34.587536 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:34.587462 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:34.930716 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:34.930687 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:35.142759 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:35.142723 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:38.343813 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:38.343781 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:34:38.514652 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:38.514621 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:39.595703 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:39.595658 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:58.423059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.423021 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd"] Jul 15 08:34:58.423468 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.423340 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6142fb3c-92f4-4e83-8896-b1eec341913b" containerName="authorino" Jul 15 08:34:58.423468 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.423351 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6142fb3c-92f4-4e83-8896-b1eec341913b" containerName="authorino" Jul 15 08:34:58.423468 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.423424 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6142fb3c-92f4-4e83-8896-b1eec341913b" containerName="authorino" Jul 15 08:34:58.426625 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.426609 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.429151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.429127 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2082f65c\"/\"openshift-service-ca.crt\"" Jul 15 08:34:58.430008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.429990 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2082f65c\"/\"default-dockercfg-n5wrd\"" Jul 15 08:34:58.430079 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.429994 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2082f65c\"/\"kube-root-ca.crt\"" Jul 15 08:34:58.430079 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.430053 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2082f65c\"/\"e2e-lifecycle-model-2082f65c-kserve-self-signed-certs\"" Jul 15 08:34:58.435866 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.435846 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd"] Jul 15 08:34:58.499136 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.499113 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-home\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.499269 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.499147 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwf9\" (UniqueName: \"kubernetes.io/projected/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-kube-api-access-gpwf9\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.499269 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.499170 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tmp-dir\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.499269 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.499204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-dshm\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.499269 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.499261 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-model-cache\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.499477 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.499334 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tls-certs\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.525184 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.525164 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:34:58.600288 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-model-cache\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.600375 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tls-certs\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.600530 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-home\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.600590 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwf9\" (UniqueName: \"kubernetes.io/projected/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-kube-api-access-gpwf9\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.600627 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tmp-dir\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.600672 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-dshm\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.600740 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-model-cache\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.600818 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-home\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.600935 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.600917 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tmp-dir\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.602691 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.602670 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-dshm\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.602852 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.602836 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tls-certs\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.608327 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.608309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwf9\" (UniqueName: \"kubernetes.io/projected/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-kube-api-access-gpwf9\") pod \"e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.736749 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.736694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:34:58.867230 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:58.867204 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd"] Jul 15 08:34:58.869567 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:34:58.869537 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b0dc9b1_eb0d_416b_80dd_69dea61833bb.slice/crio-859c3f604b373945ab9c0e6a7b440d39bccc95011033012c21c3bde8bd2e9ab2 WatchSource:0}: Error finding container 859c3f604b373945ab9c0e6a7b440d39bccc95011033012c21c3bde8bd2e9ab2: Status 404 returned error can't find the container with id 859c3f604b373945ab9c0e6a7b440d39bccc95011033012c21c3bde8bd2e9ab2 Jul 15 08:34:59.722490 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:59.722429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" event={"ID":"1b0dc9b1-eb0d-416b-80dd-69dea61833bb","Type":"ContainerStarted","Data":"459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9"} Jul 15 08:34:59.722490 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:59.722490 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" event={"ID":"1b0dc9b1-eb0d-416b-80dd-69dea61833bb","Type":"ContainerStarted","Data":"859c3f604b373945ab9c0e6a7b440d39bccc95011033012c21c3bde8bd2e9ab2"} Jul 15 08:34:59.740748 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:34:59.740703 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" podStartSLOduration=1.7406853899999999 podStartE2EDuration="1.74068539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:34:59.739468035 +0000 UTC m=+2057.255264869" watchObservedRunningTime="2026-07-15 08:34:59.74068539 +0000 UTC m=+2057.256482228" Jul 15 08:35:01.825821 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:01.825790 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:35:08.737743 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:08.737649 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:35:08.737743 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:08.737717 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:35:08.750253 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:08.750224 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:35:08.775034 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:08.775010 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:35:14.243152 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:14.243117 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:35:27.825306 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:27.825270 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:35:31.723400 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:31.723356 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:35:32.169295 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:32.169258 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:35:37.155455 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:37.155394 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd"] Jul 15 08:35:37.245507 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:37.245474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:35:43.101733 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:43.101702 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:35:43.108989 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:43.108970 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:35:52.176681 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.176625 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" podUID="1b0dc9b1-eb0d-416b-80dd-69dea61833bb" containerName="main" containerID="cri-o://459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9" gracePeriod=45 Jul 15 08:35:52.421354 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.421329 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:35:52.488006 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.487942 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-model-cache\") pod \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " Jul 15 08:35:52.488006 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.487975 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tls-certs\") pod \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " Jul 15 08:35:52.488006 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.487993 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-home\") pod \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " Jul 15 08:35:52.488259 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.488025 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpwf9\" (UniqueName: \"kubernetes.io/projected/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-kube-api-access-gpwf9\") pod \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " Jul 15 08:35:52.488259 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.488151 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-dshm\") pod \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " Jul 15 08:35:52.488259 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.488221 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tmp-dir\") pod \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\" (UID: \"1b0dc9b1-eb0d-416b-80dd-69dea61833bb\") " Jul 15 08:35:52.488259 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.488238 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b0dc9b1-eb0d-416b-80dd-69dea61833bb" (UID: "1b0dc9b1-eb0d-416b-80dd-69dea61833bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:35:52.488516 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.488487 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b0dc9b1-eb0d-416b-80dd-69dea61833bb" (UID: "1b0dc9b1-eb0d-416b-80dd-69dea61833bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:35:52.488583 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.488535 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-home" (OuterVolumeSpecName: "home") pod "1b0dc9b1-eb0d-416b-80dd-69dea61833bb" (UID: "1b0dc9b1-eb0d-416b-80dd-69dea61833bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:35:52.488583 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.488544 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:35:52.490299 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.490267 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b0dc9b1-eb0d-416b-80dd-69dea61833bb" (UID: "1b0dc9b1-eb0d-416b-80dd-69dea61833bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:35:52.490299 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.490277 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-dshm" (OuterVolumeSpecName: "dshm") pod "1b0dc9b1-eb0d-416b-80dd-69dea61833bb" (UID: "1b0dc9b1-eb0d-416b-80dd-69dea61833bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:35:52.490500 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.490327 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-kube-api-access-gpwf9" (OuterVolumeSpecName: "kube-api-access-gpwf9") pod "1b0dc9b1-eb0d-416b-80dd-69dea61833bb" (UID: "1b0dc9b1-eb0d-416b-80dd-69dea61833bb"). InnerVolumeSpecName "kube-api-access-gpwf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:35:52.589916 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.589889 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:35:52.589916 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.589912 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:35:52.590059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.589922 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwf9\" (UniqueName: \"kubernetes.io/projected/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-kube-api-access-gpwf9\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:35:52.590059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.589931 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:35:52.590059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.589939 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0dc9b1-eb0d-416b-80dd-69dea61833bb-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:35:52.986021 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.985987 2567 generic.go:356] "Generic (PLEG): container finished" podID="1b0dc9b1-eb0d-416b-80dd-69dea61833bb" containerID="459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9" exitCode=0 Jul 15 08:35:52.986377 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.986036 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" Jul 15 08:35:52.986377 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.986079 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" event={"ID":"1b0dc9b1-eb0d-416b-80dd-69dea61833bb","Type":"ContainerDied","Data":"459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9"} Jul 15 08:35:52.986377 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.986133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd" event={"ID":"1b0dc9b1-eb0d-416b-80dd-69dea61833bb","Type":"ContainerDied","Data":"859c3f604b373945ab9c0e6a7b440d39bccc95011033012c21c3bde8bd2e9ab2"} Jul 15 08:35:52.986377 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.986156 2567 scope.go:117] "RemoveContainer" containerID="459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9" Jul 15 08:35:52.995588 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.995574 2567 scope.go:117] "RemoveContainer" containerID="459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9" Jul 15 08:35:52.995814 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:35:52.995792 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9\": container with ID starting with 459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9 not found: ID does not exist" containerID="459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9" Jul 15 08:35:52.995897 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:52.995821 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9"} err="failed to get container status \"459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9\": rpc error: code = NotFound desc = could not find container \"459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9\": container with ID starting with 459fdfff76a9fa79aa046c39c19032ea64a51b036cbca473895b091ca5909bf9 not found: ID does not exist" Jul 15 08:35:53.008166 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:53.008145 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd"] Jul 15 08:35:53.009980 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:53.009961 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2082f65c/e2e-lifecycle-model-2082f65c-kserve-586d86576c-4lghd"] Jul 15 08:35:53.033303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:35:53.033280 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b0dc9b1-eb0d-416b-80dd-69dea61833bb" path="/var/lib/kubelet/pods/1b0dc9b1-eb0d-416b-80dd-69dea61833bb/volumes" Jul 15 08:36:13.913538 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:13.913506 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:15.654695 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:15.654662 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:21.955334 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:21.955302 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:28.164206 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:28.164151 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:28.269173 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:28.269147 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:36.094420 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:36.094343 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:37.181671 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:37.181635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:37.285535 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:37.285504 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:37.389915 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:37.389888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:37.478071 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:37.477996 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:38.176063 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:38.176035 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:38.273553 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:38.273522 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:38.374420 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:38.374392 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:39.960740 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:39.960709 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:41.331091 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:41.331059 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:36:46.649091 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:46.649059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:46.771571 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:46.771543 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:47.390024 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:47.389990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:47.490342 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:47.490311 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:53.831111 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:53.831067 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:53.937675 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:53.937649 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:54.032423 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:54.032396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:36:54.337495 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:36:54.337461 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:37:04.607513 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:04.607480 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:04.985453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:04.985354 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:26.027346 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:26.027311 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:32.175102 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:32.175021 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:32.277897 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:32.277870 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:32.541207 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:32.541136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:32.867333 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:32.867292 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:33.319055 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:33.318973 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:35.885634 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:35.885597 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:35.981071 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:35.981045 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:36.086849 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:36.086817 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:36.264665 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:36.264598 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:36.649573 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:36.649544 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:36.749640 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:36.749613 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:40.432558 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:40.432523 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:40.796470 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:40.796364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:41.099764 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:41.099738 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:41.402023 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:41.401932 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:43.687077 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:43.687043 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:37:43.969563 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:43.969481 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:44.066659 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:44.066625 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:44.166636 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:44.166604 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:44.645272 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:44.645235 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:44.856855 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:44.856826 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:44.957710 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:44.957642 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:48.506758 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:48.506722 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:57.557218 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:57.557180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:57.657632 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:57.657596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:37:58.337749 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:37:58.337713 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:38:04.795679 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.795642 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6"] Jul 15 08:38:04.796865 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.796838 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0dc9b1-eb0d-416b-80dd-69dea61833bb" containerName="main" Jul 15 08:38:04.797018 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.797008 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0dc9b1-eb0d-416b-80dd-69dea61833bb" containerName="main" Jul 15 08:38:04.797243 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.797231 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b0dc9b1-eb0d-416b-80dd-69dea61833bb" containerName="main" Jul 15 08:38:04.801206 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.801181 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:04.809500 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.809482 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f6b673\"/\"kube-root-ca.crt\"" Jul 15 08:38:04.809597 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.809563 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f6b673\"/\"openshift-service-ca.crt\"" Jul 15 08:38:04.809639 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.809600 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f6b673\"/\"default-dockercfg-k4c9f\"" Jul 15 08:38:04.809809 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.809795 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f6b673\"/\"test-model-f6b673-kserve-self-signed-certs\"" Jul 15 08:38:04.829265 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.829247 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6"] Jul 15 08:38:04.854065 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.854017 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:38:04.958801 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.958777 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-dshm\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:04.958902 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.958820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-home\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:04.958902 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.958841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tls-certs\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:04.958989 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.958922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-model-cache\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:04.958989 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.958947 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tmp-dir\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:04.958989 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:04.958980 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk59w\" (UniqueName: \"kubernetes.io/projected/c7625e8c-192c-44fe-a7c1-b36635eb0df5-kube-api-access-tk59w\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.059765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.059711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-dshm\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.059765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.059746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-home\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.059765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.059765 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tls-certs\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.060019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.059787 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-model-cache\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.060019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.059805 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tmp-dir\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.060019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.059837 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk59w\" (UniqueName: \"kubernetes.io/projected/c7625e8c-192c-44fe-a7c1-b36635eb0df5-kube-api-access-tk59w\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.060202 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.060088 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-model-cache\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.060202 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.060155 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tmp-dir\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.060287 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.060207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-home\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.061973 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.061953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-dshm\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.062337 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.062320 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tls-certs\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.069525 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.069505 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk59w\" (UniqueName: \"kubernetes.io/projected/c7625e8c-192c-44fe-a7c1-b36635eb0df5-kube-api-access-tk59w\") pod \"test-model-f6b673-kserve-54ccdfdb6f-sjpr6\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.110291 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.110266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:05.276010 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.275983 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6"] Jul 15 08:38:05.276749 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:38:05.276724 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7625e8c_192c_44fe_a7c1_b36635eb0df5.slice/crio-43bc9756f1479de968e8f0bdb6cded8aa2573d5e2be0861dd0bf27b91ffa6ad4 WatchSource:0}: Error finding container 43bc9756f1479de968e8f0bdb6cded8aa2573d5e2be0861dd0bf27b91ffa6ad4: Status 404 returned error can't find the container with id 43bc9756f1479de968e8f0bdb6cded8aa2573d5e2be0861dd0bf27b91ffa6ad4 Jul 15 08:38:05.613040 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.612998 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" event={"ID":"c7625e8c-192c-44fe-a7c1-b36635eb0df5","Type":"ContainerStarted","Data":"f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6"} Jul 15 08:38:05.613235 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.613051 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" event={"ID":"c7625e8c-192c-44fe-a7c1-b36635eb0df5","Type":"ContainerStarted","Data":"43bc9756f1479de968e8f0bdb6cded8aa2573d5e2be0861dd0bf27b91ffa6ad4"} Jul 15 08:38:05.639528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:05.639482 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" podStartSLOduration=1.639465377 podStartE2EDuration="1.639465377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:38:05.63702338 +0000 UTC m=+2243.152820214" watchObservedRunningTime="2026-07-15 08:38:05.639465377 +0000 UTC m=+2243.155262210" Jul 15 08:38:06.030525 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:06.030380 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:38:06.139043 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:06.138947 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:38:07.760482 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:07.760430 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:38:15.111048 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:15.111008 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:15.111458 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:15.111061 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:15.123394 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:15.123369 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:15.670150 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:15.670124 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:38:21.239199 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.239167 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg"] Jul 15 08:38:21.242418 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.242400 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.246356 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.246333 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4c59e6\"/\"default-dockercfg-bmwl4\"" Jul 15 08:38:21.246771 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.246552 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4c59e6\"/\"kube-root-ca.crt\"" Jul 15 08:38:21.246771 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.246616 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4c59e6\"/\"openshift-service-ca.crt\"" Jul 15 08:38:21.246771 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.246677 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4c59e6\"/\"test-model-4c59e6-kserve-self-signed-certs\"" Jul 15 08:38:21.253566 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.253547 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg"] Jul 15 08:38:21.268450 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.268411 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tmp-dir\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.268529 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.268463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-home\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.268577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.268532 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-model-cache\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.268577 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.268559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd528\" (UniqueName: \"kubernetes.io/projected/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-kube-api-access-kd528\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.268645 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.268580 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tls-certs\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.268645 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.268626 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-dshm\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.343199 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.343166 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:38:21.369172 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-model-cache\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.369301 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369181 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd528\" (UniqueName: \"kubernetes.io/projected/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-kube-api-access-kd528\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.369301 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369209 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tls-certs\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.369301 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369244 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-dshm\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.369301 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tmp-dir\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.369555 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-home\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.369650 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-model-cache\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.369650 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369644 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tmp-dir\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.369757 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.369741 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-home\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.371449 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.371418 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-dshm\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.371737 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.371721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tls-certs\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.378367 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.378347 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd528\" (UniqueName: \"kubernetes.io/projected/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-kube-api-access-kd528\") pod \"test-model-4c59e6-kserve-758bb89589-k46bg\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.552622 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.552556 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:21.683539 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.683502 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg"] Jul 15 08:38:21.685372 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:38:21.685349 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5e6c7a7_bef4_47d5_a6db_9216a7614f01.slice/crio-a64855a4bd35da8dbe766b5ca5d1c74b4f2fa3b4c29f3da445aa9b2fd020b2cd WatchSource:0}: Error finding container a64855a4bd35da8dbe766b5ca5d1c74b4f2fa3b4c29f3da445aa9b2fd020b2cd: Status 404 returned error can't find the container with id a64855a4bd35da8dbe766b5ca5d1c74b4f2fa3b4c29f3da445aa9b2fd020b2cd Jul 15 08:38:21.955738 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:21.955709 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:38:22.694231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:22.694196 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" event={"ID":"c5e6c7a7-bef4-47d5-a6db-9216a7614f01","Type":"ContainerStarted","Data":"4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b"} Jul 15 08:38:22.694645 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:22.694244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" event={"ID":"c5e6c7a7-bef4-47d5-a6db-9216a7614f01","Type":"ContainerStarted","Data":"a64855a4bd35da8dbe766b5ca5d1c74b4f2fa3b4c29f3da445aa9b2fd020b2cd"} Jul 15 08:38:22.717612 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:22.717565 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" podStartSLOduration=1.717550491 podStartE2EDuration="1.717550491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:38:22.717257211 +0000 UTC m=+2260.233054045" watchObservedRunningTime="2026-07-15 08:38:22.717550491 +0000 UTC m=+2260.233347324" Jul 15 08:38:31.553661 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:31.553621 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:31.554190 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:31.553676 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:31.566718 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:31.566689 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:31.748708 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:31.748672 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:38:40.892150 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:40.892117 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:38:52.450710 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:52.450676 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6"] Jul 15 08:38:52.474774 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:52.474706 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:38:53.573860 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:53.573829 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg"] Jul 15 08:38:53.831774 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:53.831657 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:38:54.334661 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:54.334627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:38:54.436162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:54.436108 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:38:59.621225 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:38:59.621187 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:39:04.215513 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:04.215469 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:07.470600 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.470555 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" podUID="c7625e8c-192c-44fe-a7c1-b36635eb0df5" containerName="main" containerID="cri-o://f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6" gracePeriod=45 Jul 15 08:39:07.711976 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.711954 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:39:07.818704 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.818640 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-dshm\") pod \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " Jul 15 08:39:07.818704 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.818668 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tmp-dir\") pod \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " Jul 15 08:39:07.818886 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.818716 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-home\") pod \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " Jul 15 08:39:07.818886 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.818741 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-model-cache\") pod \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " Jul 15 08:39:07.818886 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.818770 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tls-certs\") pod \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " Jul 15 08:39:07.818886 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.818798 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk59w\" (UniqueName: \"kubernetes.io/projected/c7625e8c-192c-44fe-a7c1-b36635eb0df5-kube-api-access-tk59w\") pod \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\" (UID: \"c7625e8c-192c-44fe-a7c1-b36635eb0df5\") " Jul 15 08:39:07.819100 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.818933 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7625e8c-192c-44fe-a7c1-b36635eb0df5" (UID: "c7625e8c-192c-44fe-a7c1-b36635eb0df5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:39:07.819100 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.819038 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7625e8c-192c-44fe-a7c1-b36635eb0df5" (UID: "c7625e8c-192c-44fe-a7c1-b36635eb0df5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:39:07.819100 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.819079 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-home" (OuterVolumeSpecName: "home") pod "c7625e8c-192c-44fe-a7c1-b36635eb0df5" (UID: "c7625e8c-192c-44fe-a7c1-b36635eb0df5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:39:07.820905 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.820879 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7625e8c-192c-44fe-a7c1-b36635eb0df5-kube-api-access-tk59w" (OuterVolumeSpecName: "kube-api-access-tk59w") pod "c7625e8c-192c-44fe-a7c1-b36635eb0df5" (UID: "c7625e8c-192c-44fe-a7c1-b36635eb0df5"). InnerVolumeSpecName "kube-api-access-tk59w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:39:07.821008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.820925 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-dshm" (OuterVolumeSpecName: "dshm") pod "c7625e8c-192c-44fe-a7c1-b36635eb0df5" (UID: "c7625e8c-192c-44fe-a7c1-b36635eb0df5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:39:07.821008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.820926 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7625e8c-192c-44fe-a7c1-b36635eb0df5" (UID: "c7625e8c-192c-44fe-a7c1-b36635eb0df5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:39:07.919760 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.919731 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:07.919760 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.919758 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:07.919929 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.919772 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:07.919929 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.919784 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk59w\" (UniqueName: \"kubernetes.io/projected/c7625e8c-192c-44fe-a7c1-b36635eb0df5-kube-api-access-tk59w\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:07.919929 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.919797 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:07.919929 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.919808 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7625e8c-192c-44fe-a7c1-b36635eb0df5-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:07.928124 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.928097 2567 generic.go:356] "Generic (PLEG): container finished" podID="c7625e8c-192c-44fe-a7c1-b36635eb0df5" containerID="f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6" exitCode=0 Jul 15 08:39:07.928270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.928148 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" Jul 15 08:39:07.928270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.928181 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" event={"ID":"c7625e8c-192c-44fe-a7c1-b36635eb0df5","Type":"ContainerDied","Data":"f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6"} Jul 15 08:39:07.928270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.928224 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6" event={"ID":"c7625e8c-192c-44fe-a7c1-b36635eb0df5","Type":"ContainerDied","Data":"43bc9756f1479de968e8f0bdb6cded8aa2573d5e2be0861dd0bf27b91ffa6ad4"} Jul 15 08:39:07.928270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.928241 2567 scope.go:117] "RemoveContainer" containerID="f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6" Jul 15 08:39:07.939556 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.939538 2567 scope.go:117] "RemoveContainer" containerID="f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6" Jul 15 08:39:07.939791 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:39:07.939771 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6\": container with ID starting with f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6 not found: ID does not exist" containerID="f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6" Jul 15 08:39:07.939860 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.939802 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6"} err="failed to get container status \"f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6\": rpc error: code = NotFound desc = could not find container \"f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6\": container with ID starting with f9f16c1316a67d24bae6b885f90f57f8d5018f3e964b2463f6a2a8bcf090e2c6 not found: ID does not exist" Jul 15 08:39:07.956558 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.956526 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6"] Jul 15 08:39:07.958342 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:07.958320 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f6b673/test-model-f6b673-kserve-54ccdfdb6f-sjpr6"] Jul 15 08:39:08.593268 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.593227 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" podUID="c5e6c7a7-bef4-47d5-a6db-9216a7614f01" containerName="main" containerID="cri-o://4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b" gracePeriod=45 Jul 15 08:39:08.831839 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.831814 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:39:08.839658 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.839635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:08.927647 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.927620 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tmp-dir\") pod \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " Jul 15 08:39:08.927785 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.927652 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-model-cache\") pod \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " Jul 15 08:39:08.927785 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.927671 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tls-certs\") pod \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " Jul 15 08:39:08.927785 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.927695 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd528\" (UniqueName: \"kubernetes.io/projected/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-kube-api-access-kd528\") pod \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " Jul 15 08:39:08.927785 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.927744 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-home\") pod \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " Jul 15 08:39:08.928008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.927808 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-dshm\") pod \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\" (UID: \"c5e6c7a7-bef4-47d5-a6db-9216a7614f01\") " Jul 15 08:39:08.928008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.927886 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5e6c7a7-bef4-47d5-a6db-9216a7614f01" (UID: "c5e6c7a7-bef4-47d5-a6db-9216a7614f01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:39:08.928008 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.927918 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5e6c7a7-bef4-47d5-a6db-9216a7614f01" (UID: "c5e6c7a7-bef4-47d5-a6db-9216a7614f01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:39:08.928144 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.928045 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-home" (OuterVolumeSpecName: "home") pod "c5e6c7a7-bef4-47d5-a6db-9216a7614f01" (UID: "c5e6c7a7-bef4-47d5-a6db-9216a7614f01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:39:08.928144 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.928129 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:08.928219 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.928149 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:08.928219 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.928163 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:08.929966 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.929945 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-dshm" (OuterVolumeSpecName: "dshm") pod "c5e6c7a7-bef4-47d5-a6db-9216a7614f01" (UID: "c5e6c7a7-bef4-47d5-a6db-9216a7614f01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:39:08.930079 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.929944 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5e6c7a7-bef4-47d5-a6db-9216a7614f01" (UID: "c5e6c7a7-bef4-47d5-a6db-9216a7614f01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:39:08.930079 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.929979 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-kube-api-access-kd528" (OuterVolumeSpecName: "kube-api-access-kd528") pod "c5e6c7a7-bef4-47d5-a6db-9216a7614f01" (UID: "c5e6c7a7-bef4-47d5-a6db-9216a7614f01"). InnerVolumeSpecName "kube-api-access-kd528". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:39:08.934866 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.934840 2567 generic.go:356] "Generic (PLEG): container finished" podID="c5e6c7a7-bef4-47d5-a6db-9216a7614f01" containerID="4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b" exitCode=0 Jul 15 08:39:08.934954 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.934881 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" Jul 15 08:39:08.934954 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.934919 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" event={"ID":"c5e6c7a7-bef4-47d5-a6db-9216a7614f01","Type":"ContainerDied","Data":"4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b"} Jul 15 08:39:08.935133 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.934959 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg" event={"ID":"c5e6c7a7-bef4-47d5-a6db-9216a7614f01","Type":"ContainerDied","Data":"a64855a4bd35da8dbe766b5ca5d1c74b4f2fa3b4c29f3da445aa9b2fd020b2cd"} Jul 15 08:39:08.935133 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.934976 2567 scope.go:117] "RemoveContainer" containerID="4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b" Jul 15 08:39:08.946642 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.946627 2567 scope.go:117] "RemoveContainer" containerID="4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b" Jul 15 08:39:08.946877 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:39:08.946860 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b\": container with ID starting with 4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b not found: ID does not exist" containerID="4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b" Jul 15 08:39:08.946932 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.946883 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b"} err="failed to get container status \"4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b\": rpc error: code = NotFound desc = could not find container \"4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b\": container with ID starting with 4a01ca781ece4e9d311a47f9f2e6db4de7a4e0e37704eaa4f3624a2a3a0ee06b not found: ID does not exist" Jul 15 08:39:08.979029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.979005 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg"] Jul 15 08:39:08.982955 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:08.982934 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4c59e6/test-model-4c59e6-kserve-758bb89589-k46bg"] Jul 15 08:39:09.029270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:09.029251 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:09.029270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:09.029270 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:09.029380 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:09.029284 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd528\" (UniqueName: \"kubernetes.io/projected/c5e6c7a7-bef4-47d5-a6db-9216a7614f01-kube-api-access-kd528\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:39:09.033130 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:09.033110 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e6c7a7-bef4-47d5-a6db-9216a7614f01" path="/var/lib/kubelet/pods/c5e6c7a7-bef4-47d5-a6db-9216a7614f01/volumes" Jul 15 08:39:09.033528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:09.033514 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7625e8c-192c-44fe-a7c1-b36635eb0df5" path="/var/lib/kubelet/pods/c7625e8c-192c-44fe-a7c1-b36635eb0df5/volumes" Jul 15 08:39:11.033268 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:11.033236 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:39:15.477934 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:15.477901 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:15.593018 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:15.592990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:19.552632 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:19.552600 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:19.827126 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:19.827045 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:20.447156 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:20.447118 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:20.547781 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:20.547741 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:21.404208 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:21.404180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:21.503928 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:21.503900 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:22.150068 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:22.150038 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:22.251349 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:22.251321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:26.516895 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:26.516853 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:26.667144 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:26.667112 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:30.473967 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:30.473927 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:33.536109 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:33.536066 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:39:35.020611 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:35.020533 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:42.247491 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:42.247461 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:42.347225 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:42.347195 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:45.772433 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:45.772399 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:46.004903 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:46.004867 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:46.361639 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:46.361611 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:46.767609 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:46.767532 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:47.217759 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:47.217725 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:49.644303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:49.644267 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:49.740690 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:49.740639 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:49.909725 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:49.909628 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:50.314046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:50.313940 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:50.591280 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:50.591180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:50.700817 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:50.700785 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:54.233678 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:54.233644 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:54.424794 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:54.424760 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:39:54.756698 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:54.756669 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:55.052070 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:55.051992 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:56.029553 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:56.029519 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:57.804774 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:57.804736 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:57.921228 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:57.921200 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:58.032091 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:58.032063 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:58.257943 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:58.257918 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:58.530658 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:58.530581 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:39:58.630347 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:39:58.630316 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:02.271647 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:02.271611 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:11.267038 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:11.267008 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:40:11.441564 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:11.441530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:11.585043 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:11.584977 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:19.331950 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:19.331915 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:19.433580 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:19.433550 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:21.511951 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:21.511918 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:21.612664 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:21.612629 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:21.789430 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:21.789348 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:25.876140 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:25.876084 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:28.027927 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:28.027886 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:40:32.021918 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:32.021878 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:32.082923 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:32.082877 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:36.665706 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:36.665666 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:36.915091 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:36.915051 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:37.619425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:37.619396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:37.715973 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:37.715943 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:43.131989 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:43.131963 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:40:43.139961 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:43.139942 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:40:47.885633 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:47.885596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:51.325624 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:51.325591 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:40:52.182645 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:52.182614 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:58.526874 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:58.526846 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:40:58.627205 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:40:58.627178 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:03.097923 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:03.097890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:03.238898 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:03.238869 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:03.631385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:03.631337 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:03.990402 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:03.990323 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:04.507425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:04.507394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:09.723069 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:09.722956 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:09.810499 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:09.810473 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:09.882553 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:09.882524 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:10.310000 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:10.309964 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:10.407386 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:10.407357 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:12.830842 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:12.830800 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:41:14.274005 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:14.273972 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:14.629703 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:14.629641 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:14.961763 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:14.961688 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:15.344574 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:15.344498 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:18.058771 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:18.058733 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:18.144248 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:18.144217 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:18.229670 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:18.229638 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:18.621493 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:18.621430 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:18.740034 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:18.740002 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:18.841711 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:18.841664 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:22.602994 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:22.602956 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:24.132137 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:24.132098 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:41:26.400182 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:26.400149 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:31.201841 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:31.201805 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:31.412953 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:31.412918 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:36.251715 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:36.251685 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:36.350839 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:36.350796 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:37.018207 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.018174 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8"] Jul 15 08:41:37.018543 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.018526 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e6c7a7-bef4-47d5-a6db-9216a7614f01" containerName="main" Jul 15 08:41:37.018543 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.018544 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e6c7a7-bef4-47d5-a6db-9216a7614f01" containerName="main" Jul 15 08:41:37.018671 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.018558 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7625e8c-192c-44fe-a7c1-b36635eb0df5" containerName="main" Jul 15 08:41:37.018671 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.018563 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7625e8c-192c-44fe-a7c1-b36635eb0df5" containerName="main" Jul 15 08:41:37.018671 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.018629 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7625e8c-192c-44fe-a7c1-b36635eb0df5" containerName="main" Jul 15 08:41:37.018671 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.018636 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e6c7a7-bef4-47d5-a6db-9216a7614f01" containerName="main" Jul 15 08:41:37.021503 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.021484 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.023988 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.023967 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"openshift-service-ca.crt\"" Jul 15 08:41:37.024149 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.024130 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"kube-root-ca.crt\"" Jul 15 08:41:37.025039 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.025022 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"default-dockercfg-5x9qx\"" Jul 15 08:41:37.025119 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.025036 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"auth-test-model-8e59d0-kserve-self-signed-certs\"" Jul 15 08:41:37.036968 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.036945 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8"] Jul 15 08:41:37.043576 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.043546 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:37.139815 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.139792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-home\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.139929 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.139829 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23557a8-4b9a-48f8-b80e-9c43b9637032-tls-certs\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.139929 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.139853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-model-cache\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.140059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.139980 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-tmp-dir\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.140059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.140035 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72h2b\" (UniqueName: \"kubernetes.io/projected/c23557a8-4b9a-48f8-b80e-9c43b9637032-kube-api-access-72h2b\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.140148 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.140084 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-dshm\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.240854 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.240825 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-dshm\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.240970 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.240866 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-home\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.241032 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.240988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23557a8-4b9a-48f8-b80e-9c43b9637032-tls-certs\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.241032 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.241024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-model-cache\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.241137 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.241084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-tmp-dir\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.241137 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.241116 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72h2b\" (UniqueName: \"kubernetes.io/projected/c23557a8-4b9a-48f8-b80e-9c43b9637032-kube-api-access-72h2b\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.241315 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.241296 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-home\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.241370 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.241345 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-model-cache\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.241420 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.241404 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-tmp-dir\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.243235 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.243214 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-dshm\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.243498 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.243483 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23557a8-4b9a-48f8-b80e-9c43b9637032-tls-certs\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.249374 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.249352 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72h2b\" (UniqueName: \"kubernetes.io/projected/c23557a8-4b9a-48f8-b80e-9c43b9637032-kube-api-access-72h2b\") pod \"auth-test-model-8e59d0-kserve-8969b774c-xw9r8\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.331411 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.331349 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:37.462306 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.462277 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8"] Jul 15 08:41:37.464172 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:41:37.464145 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc23557a8_4b9a_48f8_b80e_9c43b9637032.slice/crio-d5bff742992580aad0ae7f2beb2069f18244f10e40920499af91607549178272 WatchSource:0}: Error finding container d5bff742992580aad0ae7f2beb2069f18244f10e40920499af91607549178272: Status 404 returned error can't find the container with id d5bff742992580aad0ae7f2beb2069f18244f10e40920499af91607549178272 Jul 15 08:41:37.641657 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.641624 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" event={"ID":"c23557a8-4b9a-48f8-b80e-9c43b9637032","Type":"ContainerStarted","Data":"af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06"} Jul 15 08:41:37.641810 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.641665 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" event={"ID":"c23557a8-4b9a-48f8-b80e-9c43b9637032","Type":"ContainerStarted","Data":"d5bff742992580aad0ae7f2beb2069f18244f10e40920499af91607549178272"} Jul 15 08:41:37.662055 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:37.662007 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" podStartSLOduration=1.6619899249999999 podStartE2EDuration="1.661989925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:41:37.660790471 +0000 UTC m=+2455.176587307" watchObservedRunningTime="2026-07-15 08:41:37.661989925 +0000 UTC m=+2455.177786761" Jul 15 08:41:39.714988 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:39.714935 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:39.815039 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:39.814997 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:41:40.229348 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:40.229316 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:41:47.331669 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:47.331632 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:47.332269 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:47.331709 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:47.344116 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:47.344093 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:41:47.700738 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:41:47.700704 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:42:07.136191 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.136158 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk"] Jul 15 08:42:07.139407 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.139390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.141911 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.141890 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"kube-root-ca.crt\"" Jul 15 08:42:07.142790 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.142774 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"auth-test-model-1db148-kserve-self-signed-certs\"" Jul 15 08:42:07.143122 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.143106 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"openshift-service-ca.crt\"" Jul 15 08:42:07.143177 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.143137 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"default-dockercfg-ch6p8\"" Jul 15 08:42:07.154934 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.154907 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk"] Jul 15 08:42:07.161806 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.161781 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:42:07.166838 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.166817 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-dshm\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.166933 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.166853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-model-cache\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.166933 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.166877 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6xg\" (UniqueName: \"kubernetes.io/projected/aeac5810-ce68-4631-9248-84181d9e2f2c-kube-api-access-tv6xg\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.167054 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.166962 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-tmp-dir\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.167054 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.166989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-home\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.167054 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.167024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac5810-ce68-4631-9248-84181d9e2f2c-tls-certs\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268057 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-dshm\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268178 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-model-cache\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268178 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268090 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6xg\" (UniqueName: \"kubernetes.io/projected/aeac5810-ce68-4631-9248-84181d9e2f2c-kube-api-access-tv6xg\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268178 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-tmp-dir\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268344 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-home\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268344 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac5810-ce68-4631-9248-84181d9e2f2c-tls-certs\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268537 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-model-cache\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268610 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268545 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-tmp-dir\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.268610 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.268584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-home\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.270193 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.270172 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-dshm\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.270507 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.270491 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac5810-ce68-4631-9248-84181d9e2f2c-tls-certs\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.277508 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.277469 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6xg\" (UniqueName: \"kubernetes.io/projected/aeac5810-ce68-4631-9248-84181d9e2f2c-kube-api-access-tv6xg\") pod \"auth-test-model-1db148-kserve-84c5dd895-whjhk\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.451987 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.451924 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:07.581896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.581867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk"] Jul 15 08:42:07.583352 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:42:07.583327 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeac5810_ce68_4631_9248_84181d9e2f2c.slice/crio-b177535c598c719fe76f59a6519ef04c2505df1c846320f2dbc80108c82589e6 WatchSource:0}: Error finding container b177535c598c719fe76f59a6519ef04c2505df1c846320f2dbc80108c82589e6: Status 404 returned error can't find the container with id b177535c598c719fe76f59a6519ef04c2505df1c846320f2dbc80108c82589e6 Jul 15 08:42:07.795113 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.795021 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" event={"ID":"aeac5810-ce68-4631-9248-84181d9e2f2c","Type":"ContainerStarted","Data":"edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599"} Jul 15 08:42:07.795113 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.795072 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" event={"ID":"aeac5810-ce68-4631-9248-84181d9e2f2c","Type":"ContainerStarted","Data":"b177535c598c719fe76f59a6519ef04c2505df1c846320f2dbc80108c82589e6"} Jul 15 08:42:07.817021 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:07.816976 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" podStartSLOduration=0.816961545 podStartE2EDuration="816.961545ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:42:07.81497692 +0000 UTC m=+2485.330773767" watchObservedRunningTime="2026-07-15 08:42:07.816961545 +0000 UTC m=+2485.332758357" Jul 15 08:42:10.730155 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:10.730123 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:42:17.452045 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:17.452003 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:17.452543 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:17.452060 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:17.465126 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:17.465100 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:17.859166 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:17.859135 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:42:25.327284 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:25.327248 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:42:42.529331 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:42.529287 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:42:57.252707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:42:57.252669 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:43:14.137090 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:43:14.137061 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:43:18.700829 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:43:18.700739 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:43:32.721239 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:43:32.721206 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:43:51.922228 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:43:51.922192 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:44:09.426556 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:09.426477 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:44:12.923478 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:12.923427 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8"] Jul 15 08:44:12.944019 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:12.943988 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:44:13.941547 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:13.941515 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk"] Jul 15 08:44:14.211666 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:14.211533 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:44:22.236171 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.236131 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r"] Jul 15 08:44:22.240095 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.240077 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.242771 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.242750 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"sub-test-model-8e59d0-kserve-self-signed-certs\"" Jul 15 08:44:22.249157 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.249135 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r"] Jul 15 08:44:22.264550 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.264522 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:44:22.385707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.385680 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:44:22.387723 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.387706 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkk5\" (UniqueName: \"kubernetes.io/projected/303d9a42-27c6-4415-a108-006a89f631e7-kube-api-access-xlkk5\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.387789 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.387741 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-dshm\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.387789 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.387764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-tmp-dir\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.387862 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.387834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-home\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.387895 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.387874 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.387936 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.387918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-model-cache\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.488572 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.488517 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-model-cache\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.488572 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.488565 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkk5\" (UniqueName: \"kubernetes.io/projected/303d9a42-27c6-4415-a108-006a89f631e7-kube-api-access-xlkk5\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.488713 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.488596 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-dshm\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.488713 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.488615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-tmp-dir\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.488713 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.488634 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-home\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.488713 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.488648 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.488958 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.488937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-model-cache\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.489005 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.488980 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-tmp-dir\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.489040 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.489002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-home\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.490925 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.490902 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-dshm\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.491169 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.491152 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.503470 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.503432 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkk5\" (UniqueName: \"kubernetes.io/projected/303d9a42-27c6-4415-a108-006a89f631e7-kube-api-access-xlkk5\") pod \"sub-test-model-8e59d0-kserve-5769bc876f-2l74r\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.551497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.551468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:22.682018 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:22.681985 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r"] Jul 15 08:44:22.685014 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:44:22.684989 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod303d9a42_27c6_4415_a108_006a89f631e7.slice/crio-2d12760b5aec89c265f56da21d86f7e30042e0c19d11b907b3f2144dec6cc55a WatchSource:0}: Error finding container 2d12760b5aec89c265f56da21d86f7e30042e0c19d11b907b3f2144dec6cc55a: Status 404 returned error can't find the container with id 2d12760b5aec89c265f56da21d86f7e30042e0c19d11b907b3f2144dec6cc55a Jul 15 08:44:23.447890 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:23.447851 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" event={"ID":"303d9a42-27c6-4415-a108-006a89f631e7","Type":"ContainerStarted","Data":"b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8"} Jul 15 08:44:23.448302 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:23.447903 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" event={"ID":"303d9a42-27c6-4415-a108-006a89f631e7","Type":"ContainerStarted","Data":"2d12760b5aec89c265f56da21d86f7e30042e0c19d11b907b3f2144dec6cc55a"} Jul 15 08:44:23.468716 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:23.468669 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" podStartSLOduration=1.468655837 podStartE2EDuration="1.468655837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:44:23.466506739 +0000 UTC m=+2620.982303587" watchObservedRunningTime="2026-07-15 08:44:23.468655837 +0000 UTC m=+2620.984452670" Jul 15 08:44:27.631553 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:27.631517 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:44:27.943163 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:27.943069 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" podUID="c23557a8-4b9a-48f8-b80e-9c43b9637032" containerName="main" containerID="cri-o://af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06" gracePeriod=45 Jul 15 08:44:28.187691 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.187667 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:44:28.334846 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.334772 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-home\") pod \"c23557a8-4b9a-48f8-b80e-9c43b9637032\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " Jul 15 08:44:28.334846 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.334836 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-dshm\") pod \"c23557a8-4b9a-48f8-b80e-9c43b9637032\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " Jul 15 08:44:28.335057 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.334885 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-model-cache\") pod \"c23557a8-4b9a-48f8-b80e-9c43b9637032\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " Jul 15 08:44:28.335057 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.334918 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72h2b\" (UniqueName: \"kubernetes.io/projected/c23557a8-4b9a-48f8-b80e-9c43b9637032-kube-api-access-72h2b\") pod \"c23557a8-4b9a-48f8-b80e-9c43b9637032\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " Jul 15 08:44:28.335057 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.334943 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23557a8-4b9a-48f8-b80e-9c43b9637032-tls-certs\") pod \"c23557a8-4b9a-48f8-b80e-9c43b9637032\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " Jul 15 08:44:28.335057 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.334960 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-tmp-dir\") pod \"c23557a8-4b9a-48f8-b80e-9c43b9637032\" (UID: \"c23557a8-4b9a-48f8-b80e-9c43b9637032\") " Jul 15 08:44:28.335240 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.335047 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-home" (OuterVolumeSpecName: "home") pod "c23557a8-4b9a-48f8-b80e-9c43b9637032" (UID: "c23557a8-4b9a-48f8-b80e-9c43b9637032"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:44:28.335240 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.335127 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-model-cache" (OuterVolumeSpecName: "model-cache") pod "c23557a8-4b9a-48f8-b80e-9c43b9637032" (UID: "c23557a8-4b9a-48f8-b80e-9c43b9637032"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:44:28.335240 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.335199 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:28.335389 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.335316 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c23557a8-4b9a-48f8-b80e-9c43b9637032" (UID: "c23557a8-4b9a-48f8-b80e-9c43b9637032"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:44:28.337097 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.337070 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-dshm" (OuterVolumeSpecName: "dshm") pod "c23557a8-4b9a-48f8-b80e-9c43b9637032" (UID: "c23557a8-4b9a-48f8-b80e-9c43b9637032"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:44:28.337200 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.337134 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23557a8-4b9a-48f8-b80e-9c43b9637032-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c23557a8-4b9a-48f8-b80e-9c43b9637032" (UID: "c23557a8-4b9a-48f8-b80e-9c43b9637032"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:44:28.337200 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.337149 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23557a8-4b9a-48f8-b80e-9c43b9637032-kube-api-access-72h2b" (OuterVolumeSpecName: "kube-api-access-72h2b") pod "c23557a8-4b9a-48f8-b80e-9c43b9637032" (UID: "c23557a8-4b9a-48f8-b80e-9c43b9637032"). InnerVolumeSpecName "kube-api-access-72h2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:44:28.435701 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.435674 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72h2b\" (UniqueName: \"kubernetes.io/projected/c23557a8-4b9a-48f8-b80e-9c43b9637032-kube-api-access-72h2b\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:28.435701 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.435698 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23557a8-4b9a-48f8-b80e-9c43b9637032-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:28.435844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.435708 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:28.435844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.435716 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:28.435844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.435724 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c23557a8-4b9a-48f8-b80e-9c43b9637032-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:28.474413 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.474387 2567 generic.go:356] "Generic (PLEG): container finished" podID="c23557a8-4b9a-48f8-b80e-9c43b9637032" containerID="af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06" exitCode=0 Jul 15 08:44:28.474528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.474419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" event={"ID":"c23557a8-4b9a-48f8-b80e-9c43b9637032","Type":"ContainerDied","Data":"af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06"} Jul 15 08:44:28.474528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.474429 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" Jul 15 08:44:28.474528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.474459 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8" event={"ID":"c23557a8-4b9a-48f8-b80e-9c43b9637032","Type":"ContainerDied","Data":"d5bff742992580aad0ae7f2beb2069f18244f10e40920499af91607549178272"} Jul 15 08:44:28.474528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.474477 2567 scope.go:117] "RemoveContainer" containerID="af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06" Jul 15 08:44:28.483244 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.483229 2567 scope.go:117] "RemoveContainer" containerID="af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06" Jul 15 08:44:28.483512 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:44:28.483493 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06\": container with ID starting with af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06 not found: ID does not exist" containerID="af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06" Jul 15 08:44:28.483603 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.483519 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06"} err="failed to get container status \"af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06\": rpc error: code = NotFound desc = could not find container \"af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06\": container with ID starting with af529e1510c5edc0d5229888759a5a11a217abe78ab66e49e637d44a99de6b06 not found: ID does not exist" Jul 15 08:44:28.495142 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.495120 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8"] Jul 15 08:44:28.501086 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.501066 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/auth-test-model-8e59d0-kserve-8969b774c-xw9r8"] Jul 15 08:44:28.961085 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:28.961051 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" podUID="aeac5810-ce68-4631-9248-84181d9e2f2c" containerName="main" containerID="cri-o://edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599" gracePeriod=45 Jul 15 08:44:29.034184 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.034159 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c23557a8-4b9a-48f8-b80e-9c43b9637032" path="/var/lib/kubelet/pods/c23557a8-4b9a-48f8-b80e-9c43b9637032/volumes" Jul 15 08:44:29.210878 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.210855 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:44:29.344231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344165 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-tmp-dir\") pod \"aeac5810-ce68-4631-9248-84181d9e2f2c\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " Jul 15 08:44:29.344231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344208 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv6xg\" (UniqueName: \"kubernetes.io/projected/aeac5810-ce68-4631-9248-84181d9e2f2c-kube-api-access-tv6xg\") pod \"aeac5810-ce68-4631-9248-84181d9e2f2c\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " Jul 15 08:44:29.344421 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344266 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-dshm\") pod \"aeac5810-ce68-4631-9248-84181d9e2f2c\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " Jul 15 08:44:29.344421 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344322 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-home\") pod \"aeac5810-ce68-4631-9248-84181d9e2f2c\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " Jul 15 08:44:29.344421 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344355 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-model-cache\") pod \"aeac5810-ce68-4631-9248-84181d9e2f2c\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " Jul 15 08:44:29.344421 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344380 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac5810-ce68-4631-9248-84181d9e2f2c-tls-certs\") pod \"aeac5810-ce68-4631-9248-84181d9e2f2c\" (UID: \"aeac5810-ce68-4631-9248-84181d9e2f2c\") " Jul 15 08:44:29.344648 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344459 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aeac5810-ce68-4631-9248-84181d9e2f2c" (UID: "aeac5810-ce68-4631-9248-84181d9e2f2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:44:29.344648 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344623 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-home" (OuterVolumeSpecName: "home") pod "aeac5810-ce68-4631-9248-84181d9e2f2c" (UID: "aeac5810-ce68-4631-9248-84181d9e2f2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:44:29.344648 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344639 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "aeac5810-ce68-4631-9248-84181d9e2f2c" (UID: "aeac5810-ce68-4631-9248-84181d9e2f2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:44:29.344772 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.344679 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:29.346458 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.346421 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeac5810-ce68-4631-9248-84181d9e2f2c-kube-api-access-tv6xg" (OuterVolumeSpecName: "kube-api-access-tv6xg") pod "aeac5810-ce68-4631-9248-84181d9e2f2c" (UID: "aeac5810-ce68-4631-9248-84181d9e2f2c"). InnerVolumeSpecName "kube-api-access-tv6xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:44:29.346582 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.346421 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-dshm" (OuterVolumeSpecName: "dshm") pod "aeac5810-ce68-4631-9248-84181d9e2f2c" (UID: "aeac5810-ce68-4631-9248-84181d9e2f2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:44:29.346582 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.346533 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeac5810-ce68-4631-9248-84181d9e2f2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aeac5810-ce68-4631-9248-84181d9e2f2c" (UID: "aeac5810-ce68-4631-9248-84181d9e2f2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:44:29.445372 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.445345 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6xg\" (UniqueName: \"kubernetes.io/projected/aeac5810-ce68-4631-9248-84181d9e2f2c-kube-api-access-tv6xg\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:29.445372 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.445371 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:29.445528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.445381 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:29.445528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.445390 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac5810-ce68-4631-9248-84181d9e2f2c-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:29.445528 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.445399 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac5810-ce68-4631-9248-84181d9e2f2c-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:44:29.481125 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.481098 2567 generic.go:356] "Generic (PLEG): container finished" podID="aeac5810-ce68-4631-9248-84181d9e2f2c" containerID="edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599" exitCode=0 Jul 15 08:44:29.481224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.481138 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" Jul 15 08:44:29.481224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.481176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" event={"ID":"aeac5810-ce68-4631-9248-84181d9e2f2c","Type":"ContainerDied","Data":"edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599"} Jul 15 08:44:29.481224 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.481209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk" event={"ID":"aeac5810-ce68-4631-9248-84181d9e2f2c","Type":"ContainerDied","Data":"b177535c598c719fe76f59a6519ef04c2505df1c846320f2dbc80108c82589e6"} Jul 15 08:44:29.481374 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.481230 2567 scope.go:117] "RemoveContainer" containerID="edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599" Jul 15 08:44:29.489273 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.489258 2567 scope.go:117] "RemoveContainer" containerID="edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599" Jul 15 08:44:29.489565 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:44:29.489548 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599\": container with ID starting with edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599 not found: ID does not exist" containerID="edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599" Jul 15 08:44:29.489617 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.489572 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599"} err="failed to get container status \"edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599\": rpc error: code = NotFound desc = could not find container \"edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599\": container with ID starting with edb923206fd8a0ef88df9d85bc14516b73a759a569ffc61c6e5f01d8053b2599 not found: ID does not exist" Jul 15 08:44:29.501709 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.501689 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk"] Jul 15 08:44:29.506255 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:29.506235 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/auth-test-model-1db148-kserve-84c5dd895-whjhk"] Jul 15 08:44:31.034822 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:31.034773 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeac5810-ce68-4631-9248-84181d9e2f2c" path="/var/lib/kubelet/pods/aeac5810-ce68-4631-9248-84181d9e2f2c/volumes" Jul 15 08:44:32.552595 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:32.552551 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:32.553065 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:32.552631 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:32.565156 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:32.565132 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:33.513633 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:33.513606 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:44:50.547368 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.547335 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl"] Jul 15 08:44:50.547780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.547662 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeac5810-ce68-4631-9248-84181d9e2f2c" containerName="main" Jul 15 08:44:50.547780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.547677 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeac5810-ce68-4631-9248-84181d9e2f2c" containerName="main" Jul 15 08:44:50.547780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.547695 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c23557a8-4b9a-48f8-b80e-9c43b9637032" containerName="main" Jul 15 08:44:50.547780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.547703 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c23557a8-4b9a-48f8-b80e-9c43b9637032" containerName="main" Jul 15 08:44:50.547931 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.547789 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c23557a8-4b9a-48f8-b80e-9c43b9637032" containerName="main" Jul 15 08:44:50.547931 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.547805 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeac5810-ce68-4631-9248-84181d9e2f2c" containerName="main" Jul 15 08:44:50.552063 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.552048 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.555469 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.555447 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"openshift-service-ca.crt\"" Jul 15 08:44:50.555469 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.555455 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"sub-test-model-1db148-kserve-self-signed-certs\"" Jul 15 08:44:50.555706 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.555482 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"kube-root-ca.crt\"" Jul 15 08:44:50.555706 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.555455 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"default-dockercfg-ch6p8\"" Jul 15 08:44:50.560527 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.560507 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl"] Jul 15 08:44:50.632632 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.632605 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:44:50.704007 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.703985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-tmp-dir\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.704007 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.704013 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-home\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.704160 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.704033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b765c-28c0-4711-bbaa-bc76af82223c-tls-certs\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.704160 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.704060 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-model-cache\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.704160 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.704092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-dshm\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.704160 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.704125 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rzr\" (UniqueName: \"kubernetes.io/projected/7c3b765c-28c0-4711-bbaa-bc76af82223c-kube-api-access-44rzr\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.804689 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.804615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rzr\" (UniqueName: \"kubernetes.io/projected/7c3b765c-28c0-4711-bbaa-bc76af82223c-kube-api-access-44rzr\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.804689 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.804672 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-tmp-dir\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.804866 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.804692 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-home\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.804866 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.804718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b765c-28c0-4711-bbaa-bc76af82223c-tls-certs\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.804866 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.804738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-model-cache\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.804866 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.804772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-dshm\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.805111 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.805087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-tmp-dir\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.805176 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.805128 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-home\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.805176 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.805157 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-model-cache\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.806992 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.806967 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-dshm\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.807454 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.807422 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b765c-28c0-4711-bbaa-bc76af82223c-tls-certs\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.813821 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.813798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rzr\" (UniqueName: \"kubernetes.io/projected/7c3b765c-28c0-4711-bbaa-bc76af82223c-kube-api-access-44rzr\") pod \"sub-test-model-1db148-kserve-869cf45795-sl2bl\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.861573 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.861553 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:44:50.996740 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:50.996710 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl"] Jul 15 08:44:50.999537 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:44:50.999509 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c3b765c_28c0_4711_bbaa_bc76af82223c.slice/crio-c16f2dbd7908fb5f53f5674fe57ccf84e093540eede084cd7bdcf0b53af5220b WatchSource:0}: Error finding container c16f2dbd7908fb5f53f5674fe57ccf84e093540eede084cd7bdcf0b53af5220b: Status 404 returned error can't find the container with id c16f2dbd7908fb5f53f5674fe57ccf84e093540eede084cd7bdcf0b53af5220b Jul 15 08:44:51.593226 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:51.593193 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" event={"ID":"7c3b765c-28c0-4711-bbaa-bc76af82223c","Type":"ContainerStarted","Data":"4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820"} Jul 15 08:44:51.593617 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:51.593234 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" event={"ID":"7c3b765c-28c0-4711-bbaa-bc76af82223c","Type":"ContainerStarted","Data":"c16f2dbd7908fb5f53f5674fe57ccf84e093540eede084cd7bdcf0b53af5220b"} Jul 15 08:44:51.612002 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:51.611957 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" podStartSLOduration=1.611942687 podStartE2EDuration="1.611942687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:44:51.610205467 +0000 UTC m=+2649.126002302" watchObservedRunningTime="2026-07-15 08:44:51.611942687 +0000 UTC m=+2649.127739553" Jul 15 08:44:55.333551 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:44:55.333516 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:45:00.861752 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:00.861718 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:45:00.862219 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:00.862010 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:45:00.874540 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:00.874514 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:45:01.649639 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:01.649615 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:45:06.817594 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:06.817562 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:45:06.918896 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:06.918865 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:45:24.005183 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:24.005148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:45:25.047409 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:25.047371 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:45:43.168683 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:43.168652 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:45:43.177475 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:43.177453 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnbv5_bc482a89-5896-486c-ac28-9569d5b79c44/ovn-acl-logging/0.log" Jul 15 08:45:44.147511 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:44.147478 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:45:49.038278 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:49.038247 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r"] Jul 15 08:45:49.063357 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:49.063317 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:45:49.133717 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:49.133691 2567 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-self-signed-certs: secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:49.133827 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:49.133762 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs podName:303d9a42-27c6-4415-a108-006a89f631e7 nodeName:}" failed. No retries permitted until 2026-07-15 08:45:49.63374288 +0000 UTC m=+2707.149539921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs") pod "sub-test-model-8e59d0-kserve-5769bc876f-2l74r" (UID: "303d9a42-27c6-4415-a108-006a89f631e7") : secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:49.638572 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:49.638539 2567 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-self-signed-certs: secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:49.638762 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:49.638623 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs podName:303d9a42-27c6-4415-a108-006a89f631e7 nodeName:}" failed. No retries permitted until 2026-07-15 08:45:50.638604128 +0000 UTC m=+2708.154400939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs") pod "sub-test-model-8e59d0-kserve-5769bc876f-2l74r" (UID: "303d9a42-27c6-4415-a108-006a89f631e7") : secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:49.965601 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:49.965523 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl"] Jul 15 08:45:49.973634 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:45:49.973599 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:45:50.647380 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:50.647339 2567 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-self-signed-certs: secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:50.647765 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:50.647420 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs podName:303d9a42-27c6-4415-a108-006a89f631e7 nodeName:}" failed. No retries permitted until 2026-07-15 08:45:52.647404487 +0000 UTC m=+2710.163201298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs") pod "sub-test-model-8e59d0-kserve-5769bc876f-2l74r" (UID: "303d9a42-27c6-4415-a108-006a89f631e7") : secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:52.661054 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:52.661019 2567 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-self-signed-certs: secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:52.661476 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:52.661101 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs podName:303d9a42-27c6-4415-a108-006a89f631e7 nodeName:}" failed. No retries permitted until 2026-07-15 08:45:56.661083416 +0000 UTC m=+2714.176880232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs") pod "sub-test-model-8e59d0-kserve-5769bc876f-2l74r" (UID: "303d9a42-27c6-4415-a108-006a89f631e7") : secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:56.693628 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:56.693583 2567 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-self-signed-certs: secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:45:56.694111 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:45:56.693673 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs podName:303d9a42-27c6-4415-a108-006a89f631e7 nodeName:}" failed. No retries permitted until 2026-07-15 08:46:04.693657837 +0000 UTC m=+2722.209454649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs") pod "sub-test-model-8e59d0-kserve-5769bc876f-2l74r" (UID: "303d9a42-27c6-4415-a108-006a89f631e7") : secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:46:01.545284 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:01.545250 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:46:04.754630 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:46:04.754596 2567 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-self-signed-certs: secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:46:04.755042 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:46:04.754672 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs podName:303d9a42-27c6-4415-a108-006a89f631e7 nodeName:}" failed. No retries permitted until 2026-07-15 08:46:20.754655037 +0000 UTC m=+2738.270451848 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs") pod "sub-test-model-8e59d0-kserve-5769bc876f-2l74r" (UID: "303d9a42-27c6-4415-a108-006a89f631e7") : secret "sub-test-model-8e59d0-kserve-self-signed-certs" not found Jul 15 08:46:04.886050 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:04.886012 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" podUID="303d9a42-27c6-4415-a108-006a89f631e7" containerName="main" containerID="cri-o://b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8" gracePeriod=45 Jul 15 08:46:04.984753 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:04.984718 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" podUID="7c3b765c-28c0-4711-bbaa-bc76af82223c" containerName="main" containerID="cri-o://4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820" gracePeriod=45 Jul 15 08:46:05.130582 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.130559 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:46:05.235998 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.235976 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:46:05.260523 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260496 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-home\") pod \"303d9a42-27c6-4415-a108-006a89f631e7\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " Jul 15 08:46:05.260650 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260530 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlkk5\" (UniqueName: \"kubernetes.io/projected/303d9a42-27c6-4415-a108-006a89f631e7-kube-api-access-xlkk5\") pod \"303d9a42-27c6-4415-a108-006a89f631e7\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " Jul 15 08:46:05.260650 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260574 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-tmp-dir\") pod \"303d9a42-27c6-4415-a108-006a89f631e7\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " Jul 15 08:46:05.260650 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260601 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs\") pod \"303d9a42-27c6-4415-a108-006a89f631e7\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " Jul 15 08:46:05.260825 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260650 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-model-cache\") pod \"303d9a42-27c6-4415-a108-006a89f631e7\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " Jul 15 08:46:05.260825 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260770 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-home" (OuterVolumeSpecName: "home") pod "303d9a42-27c6-4415-a108-006a89f631e7" (UID: "303d9a42-27c6-4415-a108-006a89f631e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:46:05.260927 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260778 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-dshm\") pod \"303d9a42-27c6-4415-a108-006a89f631e7\" (UID: \"303d9a42-27c6-4415-a108-006a89f631e7\") " Jul 15 08:46:05.260927 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260864 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "303d9a42-27c6-4415-a108-006a89f631e7" (UID: "303d9a42-27c6-4415-a108-006a89f631e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:46:05.260927 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.260887 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "303d9a42-27c6-4415-a108-006a89f631e7" (UID: "303d9a42-27c6-4415-a108-006a89f631e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:46:05.261159 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.261140 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.261216 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.261162 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.261216 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.261177 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.262637 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.262602 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303d9a42-27c6-4415-a108-006a89f631e7-kube-api-access-xlkk5" (OuterVolumeSpecName: "kube-api-access-xlkk5") pod "303d9a42-27c6-4415-a108-006a89f631e7" (UID: "303d9a42-27c6-4415-a108-006a89f631e7"). InnerVolumeSpecName "kube-api-access-xlkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:46:05.263024 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.263005 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-dshm" (OuterVolumeSpecName: "dshm") pod "303d9a42-27c6-4415-a108-006a89f631e7" (UID: "303d9a42-27c6-4415-a108-006a89f631e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:46:05.263098 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.263027 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "303d9a42-27c6-4415-a108-006a89f631e7" (UID: "303d9a42-27c6-4415-a108-006a89f631e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:46:05.361376 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361358 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-tmp-dir\") pod \"7c3b765c-28c0-4711-bbaa-bc76af82223c\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " Jul 15 08:46:05.361497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361388 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-home\") pod \"7c3b765c-28c0-4711-bbaa-bc76af82223c\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " Jul 15 08:46:05.361497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361406 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44rzr\" (UniqueName: \"kubernetes.io/projected/7c3b765c-28c0-4711-bbaa-bc76af82223c-kube-api-access-44rzr\") pod \"7c3b765c-28c0-4711-bbaa-bc76af82223c\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " Jul 15 08:46:05.361497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361460 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-dshm\") pod \"7c3b765c-28c0-4711-bbaa-bc76af82223c\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " Jul 15 08:46:05.361497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361491 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b765c-28c0-4711-bbaa-bc76af82223c-tls-certs\") pod \"7c3b765c-28c0-4711-bbaa-bc76af82223c\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " Jul 15 08:46:05.361707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361639 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c3b765c-28c0-4711-bbaa-bc76af82223c" (UID: "7c3b765c-28c0-4711-bbaa-bc76af82223c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:46:05.361707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361683 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-model-cache\") pod \"7c3b765c-28c0-4711-bbaa-bc76af82223c\" (UID: \"7c3b765c-28c0-4711-bbaa-bc76af82223c\") " Jul 15 08:46:05.361707 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361696 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-home" (OuterVolumeSpecName: "home") pod "7c3b765c-28c0-4711-bbaa-bc76af82223c" (UID: "7c3b765c-28c0-4711-bbaa-bc76af82223c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:46:05.361924 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361905 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c3b765c-28c0-4711-bbaa-bc76af82223c" (UID: "7c3b765c-28c0-4711-bbaa-bc76af82223c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:46:05.361981 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361961 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303d9a42-27c6-4415-a108-006a89f631e7-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.361981 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361976 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlkk5\" (UniqueName: \"kubernetes.io/projected/303d9a42-27c6-4415-a108-006a89f631e7-kube-api-access-xlkk5\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.362060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361987 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.362060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.361996 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303d9a42-27c6-4415-a108-006a89f631e7-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.362060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.362007 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.363417 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.363397 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3b765c-28c0-4711-bbaa-bc76af82223c-kube-api-access-44rzr" (OuterVolumeSpecName: "kube-api-access-44rzr") pod "7c3b765c-28c0-4711-bbaa-bc76af82223c" (UID: "7c3b765c-28c0-4711-bbaa-bc76af82223c"). InnerVolumeSpecName "kube-api-access-44rzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:46:05.363750 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.363734 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-dshm" (OuterVolumeSpecName: "dshm") pod "7c3b765c-28c0-4711-bbaa-bc76af82223c" (UID: "7c3b765c-28c0-4711-bbaa-bc76af82223c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:46:05.363750 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.363745 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3b765c-28c0-4711-bbaa-bc76af82223c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c3b765c-28c0-4711-bbaa-bc76af82223c" (UID: "7c3b765c-28c0-4711-bbaa-bc76af82223c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:46:05.462627 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.462598 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44rzr\" (UniqueName: \"kubernetes.io/projected/7c3b765c-28c0-4711-bbaa-bc76af82223c-kube-api-access-44rzr\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.462627 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.462629 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.462806 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.462642 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b765c-28c0-4711-bbaa-bc76af82223c-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.462806 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.462655 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b765c-28c0-4711-bbaa-bc76af82223c-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:46:05.943844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.943817 2567 generic.go:356] "Generic (PLEG): container finished" podID="303d9a42-27c6-4415-a108-006a89f631e7" containerID="b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8" exitCode=0 Jul 15 08:46:05.944212 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.943857 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" Jul 15 08:46:05.944212 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.943915 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" event={"ID":"303d9a42-27c6-4415-a108-006a89f631e7","Type":"ContainerDied","Data":"b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8"} Jul 15 08:46:05.944212 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.943955 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r" event={"ID":"303d9a42-27c6-4415-a108-006a89f631e7","Type":"ContainerDied","Data":"2d12760b5aec89c265f56da21d86f7e30042e0c19d11b907b3f2144dec6cc55a"} Jul 15 08:46:05.944212 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.943972 2567 scope.go:117] "RemoveContainer" containerID="b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8" Jul 15 08:46:05.945341 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.945318 2567 generic.go:356] "Generic (PLEG): container finished" podID="7c3b765c-28c0-4711-bbaa-bc76af82223c" containerID="4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820" exitCode=0 Jul 15 08:46:05.945409 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.945384 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" Jul 15 08:46:05.945485 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.945410 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" event={"ID":"7c3b765c-28c0-4711-bbaa-bc76af82223c","Type":"ContainerDied","Data":"4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820"} Jul 15 08:46:05.945485 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.945462 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl" event={"ID":"7c3b765c-28c0-4711-bbaa-bc76af82223c","Type":"ContainerDied","Data":"c16f2dbd7908fb5f53f5674fe57ccf84e093540eede084cd7bdcf0b53af5220b"} Jul 15 08:46:05.955405 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.955387 2567 scope.go:117] "RemoveContainer" containerID="b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8" Jul 15 08:46:05.955775 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:46:05.955707 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8\": container with ID starting with b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8 not found: ID does not exist" containerID="b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8" Jul 15 08:46:05.955775 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.955738 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8"} err="failed to get container status \"b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8\": rpc error: code = NotFound desc = could not find container \"b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8\": container with ID starting with b4befaa1fe753c7839328d2cabfcdf4f82236e484f906bff04295afe442a63b8 not found: ID does not exist" Jul 15 08:46:05.955775 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.955762 2567 scope.go:117] "RemoveContainer" containerID="4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820" Jul 15 08:46:05.964402 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.964381 2567 scope.go:117] "RemoveContainer" containerID="4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820" Jul 15 08:46:05.964708 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:46:05.964686 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820\": container with ID starting with 4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820 not found: ID does not exist" containerID="4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820" Jul 15 08:46:05.964780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.964713 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820"} err="failed to get container status \"4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820\": rpc error: code = NotFound desc = could not find container \"4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820\": container with ID starting with 4a0cdf0bbb08e5b2c2e0652b044bfeb5e9c94b8452c594dd679595c110677820 not found: ID does not exist" Jul 15 08:46:05.975740 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.975718 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl"] Jul 15 08:46:05.979278 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.979259 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/sub-test-model-1db148-kserve-869cf45795-sl2bl"] Jul 15 08:46:05.990903 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.990881 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r"] Jul 15 08:46:05.993403 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:05.993385 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/sub-test-model-8e59d0-kserve-5769bc876f-2l74r"] Jul 15 08:46:07.033396 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:07.033366 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303d9a42-27c6-4415-a108-006a89f631e7" path="/var/lib/kubelet/pods/303d9a42-27c6-4415-a108-006a89f631e7/volumes" Jul 15 08:46:07.033754 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:07.033740 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3b765c-28c0-4711-bbaa-bc76af82223c" path="/var/lib/kubelet/pods/7c3b765c-28c0-4711-bbaa-bc76af82223c/volumes" Jul 15 08:46:12.925459 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:12.925411 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:46:28.034888 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.034855 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz"] Jul 15 08:46:28.035431 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.035349 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3b765c-28c0-4711-bbaa-bc76af82223c" containerName="main" Jul 15 08:46:28.035431 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.035372 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3b765c-28c0-4711-bbaa-bc76af82223c" containerName="main" Jul 15 08:46:28.035431 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.035388 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303d9a42-27c6-4415-a108-006a89f631e7" containerName="main" Jul 15 08:46:28.035431 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.035397 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="303d9a42-27c6-4415-a108-006a89f631e7" containerName="main" Jul 15 08:46:28.035688 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.035535 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3b765c-28c0-4711-bbaa-bc76af82223c" containerName="main" Jul 15 08:46:28.035688 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.035550 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="303d9a42-27c6-4415-a108-006a89f631e7" containerName="main" Jul 15 08:46:28.038422 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.038402 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.040900 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.040878 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"kube-root-ca.crt\"" Jul 15 08:46:28.040993 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.040886 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"openshift-service-ca.crt\"" Jul 15 08:46:28.041604 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.041589 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"default-dockercfg-5x9qx\"" Jul 15 08:46:28.041765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.041753 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e59d0\"/\"rate-test-model-8e59d0-kserve-self-signed-certs\"" Jul 15 08:46:28.058568 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.058546 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz"] Jul 15 08:46:28.078127 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.078108 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:46:28.115450 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.115418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac76c999-6407-44cc-a014-b00b928f810f-tls-certs\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.115574 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.115517 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-home\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.115574 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.115542 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-model-cache\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.115664 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.115596 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-dshm\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.115664 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.115644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-tmp-dir\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.115757 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.115668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7h5c\" (UniqueName: \"kubernetes.io/projected/ac76c999-6407-44cc-a014-b00b928f810f-kube-api-access-m7h5c\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216142 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216120 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-home\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216147 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-model-cache\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216166 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-dshm\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216231 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-tmp-dir\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216366 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7h5c\" (UniqueName: \"kubernetes.io/projected/ac76c999-6407-44cc-a014-b00b928f810f-kube-api-access-m7h5c\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216366 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216270 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac76c999-6407-44cc-a014-b00b928f810f-tls-certs\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216547 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216527 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-home\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216635 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216583 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-model-cache\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.216728 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.216711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-tmp-dir\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.218523 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.218504 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-dshm\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.218660 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.218642 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac76c999-6407-44cc-a014-b00b928f810f-tls-certs\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.225154 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.225135 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7h5c\" (UniqueName: \"kubernetes.io/projected/ac76c999-6407-44cc-a014-b00b928f810f-kube-api-access-m7h5c\") pod \"rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.347588 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.347557 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:28.474778 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:28.474755 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz"] Jul 15 08:46:28.476706 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:46:28.476681 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac76c999_6407_44cc_a014_b00b928f810f.slice/crio-6e8c799a444ec55e4c54c9d02c80116e04b5172550b729a9d2747743eb8b0d05 WatchSource:0}: Error finding container 6e8c799a444ec55e4c54c9d02c80116e04b5172550b729a9d2747743eb8b0d05: Status 404 returned error can't find the container with id 6e8c799a444ec55e4c54c9d02c80116e04b5172550b729a9d2747743eb8b0d05 Jul 15 08:46:29.066466 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:29.066411 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" event={"ID":"ac76c999-6407-44cc-a014-b00b928f810f","Type":"ContainerStarted","Data":"cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65"} Jul 15 08:46:29.066910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:29.066478 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" event={"ID":"ac76c999-6407-44cc-a014-b00b928f810f","Type":"ContainerStarted","Data":"6e8c799a444ec55e4c54c9d02c80116e04b5172550b729a9d2747743eb8b0d05"} Jul 15 08:46:29.084058 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:29.083991 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" podStartSLOduration=1.083972148 podStartE2EDuration="1.083972148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:46:29.082899703 +0000 UTC m=+2746.598696537" watchObservedRunningTime="2026-07-15 08:46:29.083972148 +0000 UTC m=+2746.599768983" Jul 15 08:46:31.967791 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:31.967761 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:46:32.195765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:32.195733 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:46:32.296405 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:32.296332 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:46:38.347972 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:38.347924 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:38.347972 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:38.347983 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:38.361162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:38.361135 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:39.135344 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:39.135312 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:46:40.677526 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:40.677486 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:46:40.892243 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:40.892211 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:46:57.940325 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:57.940286 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7"] Jul 15 08:46:57.943350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:57.943328 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:57.946962 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:57.946936 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"default-dockercfg-ch6p8\"" Jul 15 08:46:57.947059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:57.946999 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"rate-test-model-1db148-kserve-self-signed-certs\"" Jul 15 08:46:57.947059 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:57.947007 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"openshift-service-ca.crt\"" Jul 15 08:46:57.947300 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:57.947287 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1db148\"/\"kube-root-ca.crt\"" Jul 15 08:46:57.953251 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:57.953230 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7"] Jul 15 08:46:57.970784 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:57.970719 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:46:58.027132 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.027108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-tmp-dir\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.027243 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.027161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-model-cache\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.027243 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.027208 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bg7\" (UniqueName: \"kubernetes.io/projected/76b87ff9-3ffb-404d-be5f-46c21e003796-kube-api-access-w5bg7\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.027361 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.027259 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-home\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.027361 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.027297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b87ff9-3ffb-404d-be5f-46c21e003796-tls-certs\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.027361 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.027338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-dshm\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.128397 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bg7\" (UniqueName: \"kubernetes.io/projected/76b87ff9-3ffb-404d-be5f-46c21e003796-kube-api-access-w5bg7\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.128551 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-home\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.128551 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b87ff9-3ffb-404d-be5f-46c21e003796-tls-certs\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.128551 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128474 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-dshm\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.128551 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128512 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-tmp-dir\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.128765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128576 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-model-cache\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.128900 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128875 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-home\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.129001 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128934 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-model-cache\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.129067 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.128998 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-tmp-dir\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.130724 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.130705 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-dshm\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.130989 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.130973 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b87ff9-3ffb-404d-be5f-46c21e003796-tls-certs\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.137586 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.137563 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bg7\" (UniqueName: \"kubernetes.io/projected/76b87ff9-3ffb-404d-be5f-46c21e003796-kube-api-access-w5bg7\") pod \"rate-test-model-1db148-kserve-65dd4cc6dc-22qp7\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.253907 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.253851 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:46:58.388846 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:58.388823 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7"] Jul 15 08:46:58.390281 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:46:58.390251 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76b87ff9_3ffb_404d_be5f_46c21e003796.slice/crio-9f007443e78ce4169f71010b8b7efc6dc8aa37a6c8117589ec8ee3b54d9d3998 WatchSource:0}: Error finding container 9f007443e78ce4169f71010b8b7efc6dc8aa37a6c8117589ec8ee3b54d9d3998: Status 404 returned error can't find the container with id 9f007443e78ce4169f71010b8b7efc6dc8aa37a6c8117589ec8ee3b54d9d3998 Jul 15 08:46:59.216881 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:59.216845 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" event={"ID":"76b87ff9-3ffb-404d-be5f-46c21e003796","Type":"ContainerStarted","Data":"085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03"} Jul 15 08:46:59.217227 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:59.216889 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" event={"ID":"76b87ff9-3ffb-404d-be5f-46c21e003796","Type":"ContainerStarted","Data":"9f007443e78ce4169f71010b8b7efc6dc8aa37a6c8117589ec8ee3b54d9d3998"} Jul 15 08:46:59.236495 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:46:59.236425 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" podStartSLOduration=2.236407786 podStartE2EDuration="2.236407786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:46:59.234642353 +0000 UTC m=+2776.750439188" watchObservedRunningTime="2026-07-15 08:46:59.236407786 +0000 UTC m=+2776.752204623" Jul 15 08:47:02.490565 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:02.490529 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:47:08.254235 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:08.254147 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:47:08.254235 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:08.254212 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:47:08.267094 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:08.267070 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:47:09.276217 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:09.276189 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:47:32.023831 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:32.023792 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:47:46.941469 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:46.941419 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:47:52.748892 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:52.748859 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz"] Jul 15 08:47:52.766607 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:52.766570 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:47:53.616823 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:53.616792 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7"] Jul 15 08:47:53.633843 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:53.633805 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:47:54.681136 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:54.681102 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7631-139826929240880"] Jul 15 08:47:54.684295 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:54.684278 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826929240880" Jul 15 08:47:54.687220 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:54.687196 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-7ss9q\"" Jul 15 08:47:54.692092 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:54.692072 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826929240880"] Jul 15 08:47:54.857899 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:54.857868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94fb9\" (UniqueName: \"kubernetes.io/projected/ddace899-7444-46fa-a980-c7e1eda90b9c-kube-api-access-94fb9\") pod \"test-curl-7631-139826929240880\" (UID: \"ddace899-7444-46fa-a980-c7e1eda90b9c\") " pod="openshift-ingress/test-curl-7631-139826929240880" Jul 15 08:47:54.958575 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:54.958514 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94fb9\" (UniqueName: \"kubernetes.io/projected/ddace899-7444-46fa-a980-c7e1eda90b9c-kube-api-access-94fb9\") pod \"test-curl-7631-139826929240880\" (UID: \"ddace899-7444-46fa-a980-c7e1eda90b9c\") " pod="openshift-ingress/test-curl-7631-139826929240880" Jul 15 08:47:54.967222 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:54.967193 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94fb9\" (UniqueName: \"kubernetes.io/projected/ddace899-7444-46fa-a980-c7e1eda90b9c-kube-api-access-94fb9\") pod \"test-curl-7631-139826929240880\" (UID: \"ddace899-7444-46fa-a980-c7e1eda90b9c\") " pod="openshift-ingress/test-curl-7631-139826929240880" Jul 15 08:47:54.994090 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:54.994066 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826929240880" Jul 15 08:47:55.126497 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:55.126475 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826929240880"] Jul 15 08:47:55.128910 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:47:55.128879 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddace899_7444_46fa_a980_c7e1eda90b9c.slice/crio-ca2f6f7605322f070f6e99aab4cc6b669ed41a685baa33b62dcdc0e71cbdb892 WatchSource:0}: Error finding container ca2f6f7605322f070f6e99aab4cc6b669ed41a685baa33b62dcdc0e71cbdb892: Status 404 returned error can't find the container with id ca2f6f7605322f070f6e99aab4cc6b669ed41a685baa33b62dcdc0e71cbdb892 Jul 15 08:47:55.130142 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:55.130122 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:47:55.489174 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:55.489134 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826929240880" event={"ID":"ddace899-7444-46fa-a980-c7e1eda90b9c","Type":"ContainerStarted","Data":"ca2f6f7605322f070f6e99aab4cc6b669ed41a685baa33b62dcdc0e71cbdb892"} Jul 15 08:47:56.494986 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:56.494949 2567 generic.go:356] "Generic (PLEG): container finished" podID="ddace899-7444-46fa-a980-c7e1eda90b9c" containerID="258d88f42822b2b629f6aa24136494a23b2700f01c1814c78e626f419d468e27" exitCode=0 Jul 15 08:47:56.495342 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:56.495019 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826929240880" event={"ID":"ddace899-7444-46fa-a980-c7e1eda90b9c","Type":"ContainerDied","Data":"258d88f42822b2b629f6aa24136494a23b2700f01c1814c78e626f419d468e27"} Jul 15 08:47:57.638007 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.637987 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826929240880" Jul 15 08:47:57.660695 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.660676 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7631-139826929240880_ddace899-7444-46fa-a980-c7e1eda90b9c/test-curl-7631-139826929240880/0.log" Jul 15 08:47:57.687283 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.687258 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7631-139826929240880"] Jul 15 08:47:57.691029 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.691010 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7631-139826929240880"] Jul 15 08:47:57.782864 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.782778 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94fb9\" (UniqueName: \"kubernetes.io/projected/ddace899-7444-46fa-a980-c7e1eda90b9c-kube-api-access-94fb9\") pod \"ddace899-7444-46fa-a980-c7e1eda90b9c\" (UID: \"ddace899-7444-46fa-a980-c7e1eda90b9c\") " Jul 15 08:47:57.784922 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.784888 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddace899-7444-46fa-a980-c7e1eda90b9c-kube-api-access-94fb9" (OuterVolumeSpecName: "kube-api-access-94fb9") pod "ddace899-7444-46fa-a980-c7e1eda90b9c" (UID: "ddace899-7444-46fa-a980-c7e1eda90b9c"). InnerVolumeSpecName "kube-api-access-94fb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:47:57.813661 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.813636 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7631-139826929291312"] Jul 15 08:47:57.813936 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.813924 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddace899-7444-46fa-a980-c7e1eda90b9c" containerName="test-curl-7631-139826929240880" Jul 15 08:47:57.813976 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.813937 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddace899-7444-46fa-a980-c7e1eda90b9c" containerName="test-curl-7631-139826929240880" Jul 15 08:47:57.814013 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.814005 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddace899-7444-46fa-a980-c7e1eda90b9c" containerName="test-curl-7631-139826929240880" Jul 15 08:47:57.816822 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.816808 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826929291312" Jul 15 08:47:57.821345 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.821322 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826929291312"] Jul 15 08:47:57.884232 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.884205 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94fb9\" (UniqueName: \"kubernetes.io/projected/ddace899-7444-46fa-a980-c7e1eda90b9c-kube-api-access-94fb9\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:47:57.985177 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:57.985151 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km74p\" (UniqueName: \"kubernetes.io/projected/97410001-6eb9-45ac-a196-ff80a6e97dcd-kube-api-access-km74p\") pod \"test-curl-7631-139826929291312\" (UID: \"97410001-6eb9-45ac-a196-ff80a6e97dcd\") " pod="openshift-ingress/test-curl-7631-139826929291312" Jul 15 08:47:58.086321 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:58.086258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km74p\" (UniqueName: \"kubernetes.io/projected/97410001-6eb9-45ac-a196-ff80a6e97dcd-kube-api-access-km74p\") pod \"test-curl-7631-139826929291312\" (UID: \"97410001-6eb9-45ac-a196-ff80a6e97dcd\") " pod="openshift-ingress/test-curl-7631-139826929291312" Jul 15 08:47:58.095381 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:58.095354 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km74p\" (UniqueName: \"kubernetes.io/projected/97410001-6eb9-45ac-a196-ff80a6e97dcd-kube-api-access-km74p\") pod \"test-curl-7631-139826929291312\" (UID: \"97410001-6eb9-45ac-a196-ff80a6e97dcd\") " pod="openshift-ingress/test-curl-7631-139826929291312" Jul 15 08:47:58.126074 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:58.126053 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826929291312" Jul 15 08:47:58.251415 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:58.251391 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826929291312"] Jul 15 08:47:58.253347 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:47:58.253321 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97410001_6eb9_45ac_a196_ff80a6e97dcd.slice/crio-288486d8b4ff074601f4acdf92adafa52ad4d198487bafa8ec5c62e942f8f890 WatchSource:0}: Error finding container 288486d8b4ff074601f4acdf92adafa52ad4d198487bafa8ec5c62e942f8f890: Status 404 returned error can't find the container with id 288486d8b4ff074601f4acdf92adafa52ad4d198487bafa8ec5c62e942f8f890 Jul 15 08:47:58.505564 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:58.505535 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826929240880" Jul 15 08:47:58.505684 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:58.505578 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2f6f7605322f070f6e99aab4cc6b669ed41a685baa33b62dcdc0e71cbdb892" Jul 15 08:47:58.509883 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:58.509859 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826929291312" event={"ID":"97410001-6eb9-45ac-a196-ff80a6e97dcd","Type":"ContainerStarted","Data":"288486d8b4ff074601f4acdf92adafa52ad4d198487bafa8ec5c62e942f8f890"} Jul 15 08:47:58.517517 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:47:58.517490 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7631-139826929240880\" is forbidden: User \"system:node:ip-10-0-128-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-37.ec2.internal' and this object" podUID="ddace899-7444-46fa-a980-c7e1eda90b9c" pod="openshift-ingress/test-curl-7631-139826929240880" Jul 15 08:47:59.034131 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:59.034096 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddace899-7444-46fa-a980-c7e1eda90b9c" path="/var/lib/kubelet/pods/ddace899-7444-46fa-a980-c7e1eda90b9c/volumes" Jul 15 08:47:59.516992 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:59.516952 2567 generic.go:356] "Generic (PLEG): container finished" podID="97410001-6eb9-45ac-a196-ff80a6e97dcd" containerID="61ab13d0a1d71e9e153d1e1be49fcba2b12ccb96e1c2f0501bf318c55e84ebaa" exitCode=0 Jul 15 08:47:59.517172 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:47:59.517034 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826929291312" event={"ID":"97410001-6eb9-45ac-a196-ff80a6e97dcd","Type":"ContainerDied","Data":"61ab13d0a1d71e9e153d1e1be49fcba2b12ccb96e1c2f0501bf318c55e84ebaa"} Jul 15 08:48:00.661111 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.661090 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826929291312" Jul 15 08:48:00.682157 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.682137 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7631-139826929291312_97410001-6eb9-45ac-a196-ff80a6e97dcd/test-curl-7631-139826929291312/0.log" Jul 15 08:48:00.708161 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.708139 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7631-139826929291312"] Jul 15 08:48:00.711857 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.711836 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7631-139826929291312"] Jul 15 08:48:00.807694 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.807630 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km74p\" (UniqueName: \"kubernetes.io/projected/97410001-6eb9-45ac-a196-ff80a6e97dcd-kube-api-access-km74p\") pod \"97410001-6eb9-45ac-a196-ff80a6e97dcd\" (UID: \"97410001-6eb9-45ac-a196-ff80a6e97dcd\") " Jul 15 08:48:00.809562 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.809541 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97410001-6eb9-45ac-a196-ff80a6e97dcd-kube-api-access-km74p" (OuterVolumeSpecName: "kube-api-access-km74p") pod "97410001-6eb9-45ac-a196-ff80a6e97dcd" (UID: "97410001-6eb9-45ac-a196-ff80a6e97dcd"). InnerVolumeSpecName "kube-api-access-km74p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:00.882312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.882285 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:00.882622 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.882609 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97410001-6eb9-45ac-a196-ff80a6e97dcd" containerName="test-curl-7631-139826929291312" Jul 15 08:48:00.882665 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.882623 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="97410001-6eb9-45ac-a196-ff80a6e97dcd" containerName="test-curl-7631-139826929291312" Jul 15 08:48:00.882696 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.882678 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="97410001-6eb9-45ac-a196-ff80a6e97dcd" containerName="test-curl-7631-139826929291312" Jul 15 08:48:00.885490 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.885475 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:00.891138 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.891114 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:00.908384 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:00.908365 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km74p\" (UniqueName: \"kubernetes.io/projected/97410001-6eb9-45ac-a196-ff80a6e97dcd-kube-api-access-km74p\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:01.008698 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.008673 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6s2\" (UniqueName: \"kubernetes.io/projected/0ee17c9f-1b47-490e-97c8-ea4e98e0cab2-kube-api-access-vs6s2\") pod \"test-curl-7631-139826934413040\" (UID: \"0ee17c9f-1b47-490e-97c8-ea4e98e0cab2\") " pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:01.033148 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.033126 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97410001-6eb9-45ac-a196-ff80a6e97dcd" path="/var/lib/kubelet/pods/97410001-6eb9-45ac-a196-ff80a6e97dcd/volumes" Jul 15 08:48:01.110051 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.110025 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6s2\" (UniqueName: \"kubernetes.io/projected/0ee17c9f-1b47-490e-97c8-ea4e98e0cab2-kube-api-access-vs6s2\") pod \"test-curl-7631-139826934413040\" (UID: \"0ee17c9f-1b47-490e-97c8-ea4e98e0cab2\") " pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:01.118960 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.118939 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6s2\" (UniqueName: \"kubernetes.io/projected/0ee17c9f-1b47-490e-97c8-ea4e98e0cab2-kube-api-access-vs6s2\") pod \"test-curl-7631-139826934413040\" (UID: \"0ee17c9f-1b47-490e-97c8-ea4e98e0cab2\") " pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:01.194845 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.194823 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:01.320106 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.320084 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:01.321970 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:48:01.321943 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ee17c9f_1b47_490e_97c8_ea4e98e0cab2.slice/crio-f13fc351bf271c21f2ff02de37d794e989a962b0822abf585c28e2a03e391731 WatchSource:0}: Error finding container f13fc351bf271c21f2ff02de37d794e989a962b0822abf585c28e2a03e391731: Status 404 returned error can't find the container with id f13fc351bf271c21f2ff02de37d794e989a962b0822abf585c28e2a03e391731 Jul 15 08:48:01.528642 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.528609 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826934413040" event={"ID":"0ee17c9f-1b47-490e-97c8-ea4e98e0cab2","Type":"ContainerStarted","Data":"f13fc351bf271c21f2ff02de37d794e989a962b0822abf585c28e2a03e391731"} Jul 15 08:48:01.529728 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.529708 2567 scope.go:117] "RemoveContainer" containerID="61ab13d0a1d71e9e153d1e1be49fcba2b12ccb96e1c2f0501bf318c55e84ebaa" Jul 15 08:48:01.529812 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:01.529713 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826929291312" Jul 15 08:48:02.537069 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:02.537038 2567 generic.go:356] "Generic (PLEG): container finished" podID="0ee17c9f-1b47-490e-97c8-ea4e98e0cab2" containerID="ad0fb51eae90a4152edfdb03ab67e1db29e02b79653af0a0ad331bd3bcfe4a2c" exitCode=0 Jul 15 08:48:02.537526 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:02.537132 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826934413040" event={"ID":"0ee17c9f-1b47-490e-97c8-ea4e98e0cab2","Type":"ContainerDied","Data":"ad0fb51eae90a4152edfdb03ab67e1db29e02b79653af0a0ad331bd3bcfe4a2c"} Jul 15 08:48:03.680287 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.680263 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:03.701151 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.701121 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7631-139826934413040_0ee17c9f-1b47-490e-97c8-ea4e98e0cab2/test-curl-7631-139826934413040/0.log" Jul 15 08:48:03.725704 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.725673 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:03.729300 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.729269 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs6s2\" (UniqueName: \"kubernetes.io/projected/0ee17c9f-1b47-490e-97c8-ea4e98e0cab2-kube-api-access-vs6s2\") pod \"0ee17c9f-1b47-490e-97c8-ea4e98e0cab2\" (UID: \"0ee17c9f-1b47-490e-97c8-ea4e98e0cab2\") " Jul 15 08:48:03.730285 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.730263 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:03.731312 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.731281 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee17c9f-1b47-490e-97c8-ea4e98e0cab2-kube-api-access-vs6s2" (OuterVolumeSpecName: "kube-api-access-vs6s2") pod "0ee17c9f-1b47-490e-97c8-ea4e98e0cab2" (UID: "0ee17c9f-1b47-490e-97c8-ea4e98e0cab2"). InnerVolumeSpecName "kube-api-access-vs6s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:03.830166 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.830084 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs6s2\" (UniqueName: \"kubernetes.io/projected/0ee17c9f-1b47-490e-97c8-ea4e98e0cab2-kube-api-access-vs6s2\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:03.857560 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.857529 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:03.857838 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.857826 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee17c9f-1b47-490e-97c8-ea4e98e0cab2" containerName="test-curl-7631-139826934413040" Jul 15 08:48:03.857838 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.857839 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee17c9f-1b47-490e-97c8-ea4e98e0cab2" containerName="test-curl-7631-139826934413040" Jul 15 08:48:03.857923 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.857909 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee17c9f-1b47-490e-97c8-ea4e98e0cab2" containerName="test-curl-7631-139826934413040" Jul 15 08:48:03.860762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.860747 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:03.867644 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.867607 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:03.930953 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:03.930922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdlj\" (UniqueName: \"kubernetes.io/projected/48ea7b57-222d-4f25-97df-cde4b9aee2a2-kube-api-access-vxdlj\") pod \"test-curl-7631-139826934413040\" (UID: \"48ea7b57-222d-4f25-97df-cde4b9aee2a2\") " pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:04.032024 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.031987 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdlj\" (UniqueName: \"kubernetes.io/projected/48ea7b57-222d-4f25-97df-cde4b9aee2a2-kube-api-access-vxdlj\") pod \"test-curl-7631-139826934413040\" (UID: \"48ea7b57-222d-4f25-97df-cde4b9aee2a2\") " pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:04.040969 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.040946 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdlj\" (UniqueName: \"kubernetes.io/projected/48ea7b57-222d-4f25-97df-cde4b9aee2a2-kube-api-access-vxdlj\") pod \"test-curl-7631-139826934413040\" (UID: \"48ea7b57-222d-4f25-97df-cde4b9aee2a2\") " pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:04.169822 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.169797 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:04.299883 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.299858 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:04.300758 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:48:04.300732 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48ea7b57_222d_4f25_97df_cde4b9aee2a2.slice/crio-78a7506606e667dfa129e5f291f601f87c0a25d00026debdac64b493007dcd18 WatchSource:0}: Error finding container 78a7506606e667dfa129e5f291f601f87c0a25d00026debdac64b493007dcd18: Status 404 returned error can't find the container with id 78a7506606e667dfa129e5f291f601f87c0a25d00026debdac64b493007dcd18 Jul 15 08:48:04.550233 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.550180 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13fc351bf271c21f2ff02de37d794e989a962b0822abf585c28e2a03e391731" Jul 15 08:48:04.550233 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.550181 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:04.551360 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.551336 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826934413040" event={"ID":"48ea7b57-222d-4f25-97df-cde4b9aee2a2","Type":"ContainerStarted","Data":"78a7506606e667dfa129e5f291f601f87c0a25d00026debdac64b493007dcd18"} Jul 15 08:48:04.561849 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.561814 2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7631-139826934413040" oldPodUID="0ee17c9f-1b47-490e-97c8-ea4e98e0cab2" podUID="48ea7b57-222d-4f25-97df-cde4b9aee2a2" Jul 15 08:48:04.823001 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:04.822941 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:48:05.034037 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:05.034009 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee17c9f-1b47-490e-97c8-ea4e98e0cab2" path="/var/lib/kubelet/pods/0ee17c9f-1b47-490e-97c8-ea4e98e0cab2/volumes" Jul 15 08:48:05.558643 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:05.558607 2567 generic.go:356] "Generic (PLEG): container finished" podID="48ea7b57-222d-4f25-97df-cde4b9aee2a2" containerID="523b88fec8debafe89ecb105dc31afb07f0afea944bac1bfd2fdf1e46422faa9" exitCode=0 Jul 15 08:48:05.558826 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:05.558694 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826934413040" event={"ID":"48ea7b57-222d-4f25-97df-cde4b9aee2a2","Type":"ContainerDied","Data":"523b88fec8debafe89ecb105dc31afb07f0afea944bac1bfd2fdf1e46422faa9"} Jul 15 08:48:06.700724 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.700704 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:06.723726 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.723705 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7631-139826934413040_48ea7b57-222d-4f25-97df-cde4b9aee2a2/test-curl-7631-139826934413040/0.log" Jul 15 08:48:06.747820 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.747793 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:06.751745 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.751725 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxdlj\" (UniqueName: \"kubernetes.io/projected/48ea7b57-222d-4f25-97df-cde4b9aee2a2-kube-api-access-vxdlj\") pod \"48ea7b57-222d-4f25-97df-cde4b9aee2a2\" (UID: \"48ea7b57-222d-4f25-97df-cde4b9aee2a2\") " Jul 15 08:48:06.751888 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.751870 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7631-139826934413040"] Jul 15 08:48:06.753610 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.753593 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ea7b57-222d-4f25-97df-cde4b9aee2a2-kube-api-access-vxdlj" (OuterVolumeSpecName: "kube-api-access-vxdlj") pod "48ea7b57-222d-4f25-97df-cde4b9aee2a2" (UID: "48ea7b57-222d-4f25-97df-cde4b9aee2a2"). InnerVolumeSpecName "kube-api-access-vxdlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:06.853275 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.853255 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdlj\" (UniqueName: \"kubernetes.io/projected/48ea7b57-222d-4f25-97df-cde4b9aee2a2-kube-api-access-vxdlj\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:06.894712 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.894686 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:06.895020 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.894990 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ea7b57-222d-4f25-97df-cde4b9aee2a2" containerName="test-curl-7631-139826934413040" Jul 15 08:48:06.895069 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.895022 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ea7b57-222d-4f25-97df-cde4b9aee2a2" containerName="test-curl-7631-139826934413040" Jul 15 08:48:06.895113 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.895082 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ea7b57-222d-4f25-97df-cde4b9aee2a2" containerName="test-curl-7631-139826934413040" Jul 15 08:48:06.897851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.897835 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:06.907802 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.907773 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:06.954289 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:06.954260 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzk7\" (UniqueName: \"kubernetes.io/projected/b1497515-57ca-4631-8918-cc2cdbf885b7-kube-api-access-7bzk7\") pod \"test-curl-7631-139826930779456\" (UID: \"b1497515-57ca-4631-8918-cc2cdbf885b7\") " pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:07.033557 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.033532 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ea7b57-222d-4f25-97df-cde4b9aee2a2" path="/var/lib/kubelet/pods/48ea7b57-222d-4f25-97df-cde4b9aee2a2/volumes" Jul 15 08:48:07.055515 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.055490 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzk7\" (UniqueName: \"kubernetes.io/projected/b1497515-57ca-4631-8918-cc2cdbf885b7-kube-api-access-7bzk7\") pod \"test-curl-7631-139826930779456\" (UID: \"b1497515-57ca-4631-8918-cc2cdbf885b7\") " pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:07.064742 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.064722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzk7\" (UniqueName: \"kubernetes.io/projected/b1497515-57ca-4631-8918-cc2cdbf885b7-kube-api-access-7bzk7\") pod \"test-curl-7631-139826930779456\" (UID: \"b1497515-57ca-4631-8918-cc2cdbf885b7\") " pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:07.206945 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.206896 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:07.336999 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.336976 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:07.338646 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:48:07.338621 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1497515_57ca_4631_8918_cc2cdbf885b7.slice/crio-bc92298de9bfb65a7141783733ab07260ef8fcdd1309444e39822f999846a7c6 WatchSource:0}: Error finding container bc92298de9bfb65a7141783733ab07260ef8fcdd1309444e39822f999846a7c6: Status 404 returned error can't find the container with id bc92298de9bfb65a7141783733ab07260ef8fcdd1309444e39822f999846a7c6 Jul 15 08:48:07.569557 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.569491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826930779456" event={"ID":"b1497515-57ca-4631-8918-cc2cdbf885b7","Type":"ContainerStarted","Data":"bc92298de9bfb65a7141783733ab07260ef8fcdd1309444e39822f999846a7c6"} Jul 15 08:48:07.570744 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.570725 2567 scope.go:117] "RemoveContainer" containerID="523b88fec8debafe89ecb105dc31afb07f0afea944bac1bfd2fdf1e46422faa9" Jul 15 08:48:07.570853 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.570767 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826934413040" Jul 15 08:48:07.768966 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:07.768937 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" podUID="ac76c999-6407-44cc-a014-b00b928f810f" containerName="main" containerID="cri-o://cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65" gracePeriod=45 Jul 15 08:48:08.015198 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.015175 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:48:08.063927 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.063901 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-dshm\") pod \"ac76c999-6407-44cc-a014-b00b928f810f\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " Jul 15 08:48:08.064058 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.063937 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-home\") pod \"ac76c999-6407-44cc-a014-b00b928f810f\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " Jul 15 08:48:08.064058 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.063997 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-tmp-dir\") pod \"ac76c999-6407-44cc-a014-b00b928f810f\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " Jul 15 08:48:08.064058 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064036 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7h5c\" (UniqueName: \"kubernetes.io/projected/ac76c999-6407-44cc-a014-b00b928f810f-kube-api-access-m7h5c\") pod \"ac76c999-6407-44cc-a014-b00b928f810f\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " Jul 15 08:48:08.064219 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064070 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac76c999-6407-44cc-a014-b00b928f810f-tls-certs\") pod \"ac76c999-6407-44cc-a014-b00b928f810f\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " Jul 15 08:48:08.064219 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064092 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-model-cache\") pod \"ac76c999-6407-44cc-a014-b00b928f810f\" (UID: \"ac76c999-6407-44cc-a014-b00b928f810f\") " Jul 15 08:48:08.064219 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064200 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-home" (OuterVolumeSpecName: "home") pod "ac76c999-6407-44cc-a014-b00b928f810f" (UID: "ac76c999-6407-44cc-a014-b00b928f810f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:08.064383 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064324 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac76c999-6407-44cc-a014-b00b928f810f" (UID: "ac76c999-6407-44cc-a014-b00b928f810f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:08.064453 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064392 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac76c999-6407-44cc-a014-b00b928f810f" (UID: "ac76c999-6407-44cc-a014-b00b928f810f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:08.064519 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064502 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:08.064519 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064516 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:08.064625 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.064526 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:08.065910 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.065893 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-dshm" (OuterVolumeSpecName: "dshm") pod "ac76c999-6407-44cc-a014-b00b928f810f" (UID: "ac76c999-6407-44cc-a014-b00b928f810f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:08.066071 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.066053 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac76c999-6407-44cc-a014-b00b928f810f-kube-api-access-m7h5c" (OuterVolumeSpecName: "kube-api-access-m7h5c") pod "ac76c999-6407-44cc-a014-b00b928f810f" (UID: "ac76c999-6407-44cc-a014-b00b928f810f"). InnerVolumeSpecName "kube-api-access-m7h5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:08.066162 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.066049 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac76c999-6407-44cc-a014-b00b928f810f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac76c999-6407-44cc-a014-b00b928f810f" (UID: "ac76c999-6407-44cc-a014-b00b928f810f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:48:08.165805 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.165782 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac76c999-6407-44cc-a014-b00b928f810f-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:08.165805 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.165806 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac76c999-6407-44cc-a014-b00b928f810f-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:08.165948 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.165815 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7h5c\" (UniqueName: \"kubernetes.io/projected/ac76c999-6407-44cc-a014-b00b928f810f-kube-api-access-m7h5c\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:08.577021 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.576946 2567 generic.go:356] "Generic (PLEG): container finished" podID="ac76c999-6407-44cc-a014-b00b928f810f" containerID="cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65" exitCode=0 Jul 15 08:48:08.577021 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.576981 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" Jul 15 08:48:08.577215 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.577033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" event={"ID":"ac76c999-6407-44cc-a014-b00b928f810f","Type":"ContainerDied","Data":"cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65"} Jul 15 08:48:08.577215 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.577071 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz" event={"ID":"ac76c999-6407-44cc-a014-b00b928f810f","Type":"ContainerDied","Data":"6e8c799a444ec55e4c54c9d02c80116e04b5172550b729a9d2747743eb8b0d05"} Jul 15 08:48:08.577215 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.577093 2567 scope.go:117] "RemoveContainer" containerID="cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65" Jul 15 08:48:08.578648 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.578627 2567 generic.go:356] "Generic (PLEG): container finished" podID="b1497515-57ca-4631-8918-cc2cdbf885b7" containerID="9f25562a2ad53a7bee0d3b0e0b5abe0c58d3c3b0251b5c5329ca7905debd82fe" exitCode=0 Jul 15 08:48:08.578734 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.578716 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826930779456" event={"ID":"b1497515-57ca-4631-8918-cc2cdbf885b7","Type":"ContainerDied","Data":"9f25562a2ad53a7bee0d3b0e0b5abe0c58d3c3b0251b5c5329ca7905debd82fe"} Jul 15 08:48:08.586724 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.586703 2567 scope.go:117] "RemoveContainer" containerID="cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65" Jul 15 08:48:08.586995 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:48:08.586977 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65\": container with ID starting with cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65 not found: ID does not exist" containerID="cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65" Jul 15 08:48:08.587069 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.587003 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65"} err="failed to get container status \"cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65\": rpc error: code = NotFound desc = could not find container \"cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65\": container with ID starting with cf889e262b5daccfe038b501df38c003a9b25ee3f452e013f136590fe3ec9a65 not found: ID does not exist" Jul 15 08:48:08.608689 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.608667 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz"] Jul 15 08:48:08.614593 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.614573 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8e59d0/rate-test-model-8e59d0-kserve-755ddc5c5-gl7zz"] Jul 15 08:48:08.642909 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.642881 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" podUID="76b87ff9-3ffb-404d-be5f-46c21e003796" containerName="main" containerID="cri-o://085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03" gracePeriod=45 Jul 15 08:48:08.895234 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.895213 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:48:08.971731 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.971708 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-tmp-dir\") pod \"76b87ff9-3ffb-404d-be5f-46c21e003796\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " Jul 15 08:48:08.971844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.971753 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-model-cache\") pod \"76b87ff9-3ffb-404d-be5f-46c21e003796\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " Jul 15 08:48:08.971844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.971771 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5bg7\" (UniqueName: \"kubernetes.io/projected/76b87ff9-3ffb-404d-be5f-46c21e003796-kube-api-access-w5bg7\") pod \"76b87ff9-3ffb-404d-be5f-46c21e003796\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " Jul 15 08:48:08.971844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.971816 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-home\") pod \"76b87ff9-3ffb-404d-be5f-46c21e003796\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " Jul 15 08:48:08.971844 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.971840 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b87ff9-3ffb-404d-be5f-46c21e003796-tls-certs\") pod \"76b87ff9-3ffb-404d-be5f-46c21e003796\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " Jul 15 08:48:08.972048 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.971887 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-dshm\") pod \"76b87ff9-3ffb-404d-be5f-46c21e003796\" (UID: \"76b87ff9-3ffb-404d-be5f-46c21e003796\") " Jul 15 08:48:08.972048 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.971963 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-model-cache" (OuterVolumeSpecName: "model-cache") pod "76b87ff9-3ffb-404d-be5f-46c21e003796" (UID: "76b87ff9-3ffb-404d-be5f-46c21e003796"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:08.972048 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.971998 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76b87ff9-3ffb-404d-be5f-46c21e003796" (UID: "76b87ff9-3ffb-404d-be5f-46c21e003796"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:08.972205 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.972099 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-home" (OuterVolumeSpecName: "home") pod "76b87ff9-3ffb-404d-be5f-46c21e003796" (UID: "76b87ff9-3ffb-404d-be5f-46c21e003796"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:08.972205 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.972109 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:08.972205 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.972128 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:08.973902 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.973881 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-dshm" (OuterVolumeSpecName: "dshm") pod "76b87ff9-3ffb-404d-be5f-46c21e003796" (UID: "76b87ff9-3ffb-404d-be5f-46c21e003796"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:08.974004 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.973985 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b87ff9-3ffb-404d-be5f-46c21e003796-kube-api-access-w5bg7" (OuterVolumeSpecName: "kube-api-access-w5bg7") pod "76b87ff9-3ffb-404d-be5f-46c21e003796" (UID: "76b87ff9-3ffb-404d-be5f-46c21e003796"). InnerVolumeSpecName "kube-api-access-w5bg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:08.974046 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:08.974028 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b87ff9-3ffb-404d-be5f-46c21e003796-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76b87ff9-3ffb-404d-be5f-46c21e003796" (UID: "76b87ff9-3ffb-404d-be5f-46c21e003796"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:48:09.033398 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.033365 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac76c999-6407-44cc-a014-b00b928f810f" path="/var/lib/kubelet/pods/ac76c999-6407-44cc-a014-b00b928f810f/volumes" Jul 15 08:48:09.072762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.072734 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5bg7\" (UniqueName: \"kubernetes.io/projected/76b87ff9-3ffb-404d-be5f-46c21e003796-kube-api-access-w5bg7\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:09.072762 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.072758 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:09.072901 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.072773 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b87ff9-3ffb-404d-be5f-46c21e003796-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:09.072901 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.072784 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b87ff9-3ffb-404d-be5f-46c21e003796-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:09.586199 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.586170 2567 generic.go:356] "Generic (PLEG): container finished" podID="76b87ff9-3ffb-404d-be5f-46c21e003796" containerID="085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03" exitCode=0 Jul 15 08:48:09.586350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.586213 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" Jul 15 08:48:09.586350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.586254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" event={"ID":"76b87ff9-3ffb-404d-be5f-46c21e003796","Type":"ContainerDied","Data":"085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03"} Jul 15 08:48:09.586350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.586288 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7" event={"ID":"76b87ff9-3ffb-404d-be5f-46c21e003796","Type":"ContainerDied","Data":"9f007443e78ce4169f71010b8b7efc6dc8aa37a6c8117589ec8ee3b54d9d3998"} Jul 15 08:48:09.586350 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.586307 2567 scope.go:117] "RemoveContainer" containerID="085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03" Jul 15 08:48:09.598152 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.597713 2567 scope.go:117] "RemoveContainer" containerID="085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03" Jul 15 08:48:09.598152 ip-10-0-128-37 kubenswrapper[2567]: E0715 08:48:09.598076 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03\": container with ID starting with 085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03 not found: ID does not exist" containerID="085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03" Jul 15 08:48:09.598152 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.598122 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03"} err="failed to get container status \"085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03\": rpc error: code = NotFound desc = could not find container \"085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03\": container with ID starting with 085de0260cae9e2cb8289d88cbe8ffdb0c4f8639a1ed81c8ef72206924a86d03 not found: ID does not exist" Jul 15 08:48:09.621130 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.621107 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7"] Jul 15 08:48:09.622970 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.622950 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1db148/rate-test-model-1db148-kserve-65dd4cc6dc-22qp7"] Jul 15 08:48:09.723269 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.723247 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:09.744031 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.744011 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7631-139826930779456_b1497515-57ca-4631-8918-cc2cdbf885b7/test-curl-7631-139826930779456/0.log" Jul 15 08:48:09.769119 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.769095 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:09.772924 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.772905 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:09.779026 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.779007 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bzk7\" (UniqueName: \"kubernetes.io/projected/b1497515-57ca-4631-8918-cc2cdbf885b7-kube-api-access-7bzk7\") pod \"b1497515-57ca-4631-8918-cc2cdbf885b7\" (UID: \"b1497515-57ca-4631-8918-cc2cdbf885b7\") " Jul 15 08:48:09.780942 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.780916 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1497515-57ca-4631-8918-cc2cdbf885b7-kube-api-access-7bzk7" (OuterVolumeSpecName: "kube-api-access-7bzk7") pod "b1497515-57ca-4631-8918-cc2cdbf885b7" (UID: "b1497515-57ca-4631-8918-cc2cdbf885b7"). InnerVolumeSpecName "kube-api-access-7bzk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:09.880330 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.880312 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bzk7\" (UniqueName: \"kubernetes.io/projected/b1497515-57ca-4631-8918-cc2cdbf885b7-kube-api-access-7bzk7\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:09.904139 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904111 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:09.904563 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904548 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b87ff9-3ffb-404d-be5f-46c21e003796" containerName="main" Jul 15 08:48:09.904646 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904566 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b87ff9-3ffb-404d-be5f-46c21e003796" containerName="main" Jul 15 08:48:09.904646 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904598 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1497515-57ca-4631-8918-cc2cdbf885b7" containerName="test-curl-7631-139826930779456" Jul 15 08:48:09.904646 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904610 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1497515-57ca-4631-8918-cc2cdbf885b7" containerName="test-curl-7631-139826930779456" Jul 15 08:48:09.904646 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904634 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac76c999-6407-44cc-a014-b00b928f810f" containerName="main" Jul 15 08:48:09.904646 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904643 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac76c999-6407-44cc-a014-b00b928f810f" containerName="main" Jul 15 08:48:09.904881 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904718 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1497515-57ca-4631-8918-cc2cdbf885b7" containerName="test-curl-7631-139826930779456" Jul 15 08:48:09.904881 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904730 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac76c999-6407-44cc-a014-b00b928f810f" containerName="main" Jul 15 08:48:09.904881 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.904741 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b87ff9-3ffb-404d-be5f-46c21e003796" containerName="main" Jul 15 08:48:09.907612 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.907596 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:09.914804 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.914783 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:09.980719 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:09.980699 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52v8s\" (UniqueName: \"kubernetes.io/projected/78759d1f-bfe9-49e9-886a-025ae04f4490-kube-api-access-52v8s\") pod \"test-curl-7631-139826930779456\" (UID: \"78759d1f-bfe9-49e9-886a-025ae04f4490\") " pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:10.081677 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:10.081648 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52v8s\" (UniqueName: \"kubernetes.io/projected/78759d1f-bfe9-49e9-886a-025ae04f4490-kube-api-access-52v8s\") pod \"test-curl-7631-139826930779456\" (UID: \"78759d1f-bfe9-49e9-886a-025ae04f4490\") " pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:10.090890 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:10.090862 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52v8s\" (UniqueName: \"kubernetes.io/projected/78759d1f-bfe9-49e9-886a-025ae04f4490-kube-api-access-52v8s\") pod \"test-curl-7631-139826930779456\" (UID: \"78759d1f-bfe9-49e9-886a-025ae04f4490\") " pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:10.216565 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:10.216518 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:10.343047 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:10.343021 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:10.345015 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:48:10.344990 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78759d1f_bfe9_49e9_886a_025ae04f4490.slice/crio-eb205fcb1b535d601f9280d368a0004b3593157f9064180ada37bacfad1e2ff7 WatchSource:0}: Error finding container eb205fcb1b535d601f9280d368a0004b3593157f9064180ada37bacfad1e2ff7: Status 404 returned error can't find the container with id eb205fcb1b535d601f9280d368a0004b3593157f9064180ada37bacfad1e2ff7 Jul 15 08:48:10.600261 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:10.600220 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826930779456" event={"ID":"78759d1f-bfe9-49e9-886a-025ae04f4490","Type":"ContainerStarted","Data":"eb205fcb1b535d601f9280d368a0004b3593157f9064180ada37bacfad1e2ff7"} Jul 15 08:48:10.601571 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:10.601551 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc92298de9bfb65a7141783733ab07260ef8fcdd1309444e39822f999846a7c6" Jul 15 08:48:10.601571 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:10.601559 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:10.614485 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:10.614418 2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7631-139826930779456" oldPodUID="b1497515-57ca-4631-8918-cc2cdbf885b7" podUID="78759d1f-bfe9-49e9-886a-025ae04f4490" Jul 15 08:48:11.033267 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:11.033186 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b87ff9-3ffb-404d-be5f-46c21e003796" path="/var/lib/kubelet/pods/76b87ff9-3ffb-404d-be5f-46c21e003796/volumes" Jul 15 08:48:11.033636 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:11.033557 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1497515-57ca-4631-8918-cc2cdbf885b7" path="/var/lib/kubelet/pods/b1497515-57ca-4631-8918-cc2cdbf885b7/volumes" Jul 15 08:48:11.608030 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:11.607996 2567 generic.go:356] "Generic (PLEG): container finished" podID="78759d1f-bfe9-49e9-886a-025ae04f4490" containerID="50f2f195d3e1a6d3a16e6a143c8aa3608af56efc7182706b8129841c44ec2ab4" exitCode=0 Jul 15 08:48:11.608180 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:11.608052 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7631-139826930779456" event={"ID":"78759d1f-bfe9-49e9-886a-025ae04f4490","Type":"ContainerDied","Data":"50f2f195d3e1a6d3a16e6a143c8aa3608af56efc7182706b8129841c44ec2ab4"} Jul 15 08:48:12.748988 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.748965 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:12.770600 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.770564 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7631-139826930779456_78759d1f-bfe9-49e9-886a-025ae04f4490/test-curl-7631-139826930779456/0.log" Jul 15 08:48:12.796387 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.796362 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:12.800280 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.800259 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7631-139826930779456"] Jul 15 08:48:12.800506 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.800489 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52v8s\" (UniqueName: \"kubernetes.io/projected/78759d1f-bfe9-49e9-886a-025ae04f4490-kube-api-access-52v8s\") pod \"78759d1f-bfe9-49e9-886a-025ae04f4490\" (UID: \"78759d1f-bfe9-49e9-886a-025ae04f4490\") " Jul 15 08:48:12.802345 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.802320 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78759d1f-bfe9-49e9-886a-025ae04f4490-kube-api-access-52v8s" (OuterVolumeSpecName: "kube-api-access-52v8s") pod "78759d1f-bfe9-49e9-886a-025ae04f4490" (UID: "78759d1f-bfe9-49e9-886a-025ae04f4490"). InnerVolumeSpecName "kube-api-access-52v8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:12.901965 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.901914 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52v8s\" (UniqueName: \"kubernetes.io/projected/78759d1f-bfe9-49e9-886a-025ae04f4490-kube-api-access-52v8s\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:12.988371 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.988348 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj"] Jul 15 08:48:12.988678 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.988665 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78759d1f-bfe9-49e9-886a-025ae04f4490" containerName="test-curl-7631-139826930779456" Jul 15 08:48:12.988727 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.988681 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="78759d1f-bfe9-49e9-886a-025ae04f4490" containerName="test-curl-7631-139826930779456" Jul 15 08:48:12.988780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.988747 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="78759d1f-bfe9-49e9-886a-025ae04f4490" containerName="test-curl-7631-139826930779456" Jul 15 08:48:12.991649 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.991633 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" Jul 15 08:48:12.994100 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.994079 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 08:48:12.994100 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:12.994084 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:48:13.001346 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.001325 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj"] Jul 15 08:48:13.033898 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.033875 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78759d1f-bfe9-49e9-886a-025ae04f4490" path="/var/lib/kubelet/pods/78759d1f-bfe9-49e9-886a-025ae04f4490/volumes" Jul 15 08:48:13.103388 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.103363 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b147395f-8227-44aa-b43b-d6fef3bbcd99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj\" (UID: \"b147395f-8227-44aa-b43b-d6fef3bbcd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" Jul 15 08:48:13.204595 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.204541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b147395f-8227-44aa-b43b-d6fef3bbcd99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj\" (UID: \"b147395f-8227-44aa-b43b-d6fef3bbcd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" Jul 15 08:48:13.205086 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.205068 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b147395f-8227-44aa-b43b-d6fef3bbcd99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj\" (UID: \"b147395f-8227-44aa-b43b-d6fef3bbcd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" Jul 15 08:48:13.302943 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.302920 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" Jul 15 08:48:13.430325 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.430300 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj"] Jul 15 08:48:13.433067 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:48:13.433041 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb147395f_8227_44aa_b43b_d6fef3bbcd99.slice/crio-52c4bd9d8ead3b9998e43d438b4c23b6e64b840a9fc2db3df3cbf294efc90911 WatchSource:0}: Error finding container 52c4bd9d8ead3b9998e43d438b4c23b6e64b840a9fc2db3df3cbf294efc90911: Status 404 returned error can't find the container with id 52c4bd9d8ead3b9998e43d438b4c23b6e64b840a9fc2db3df3cbf294efc90911 Jul 15 08:48:13.510984 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.510925 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:13.618855 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.618829 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" event={"ID":"b147395f-8227-44aa-b43b-d6fef3bbcd99","Type":"ContainerStarted","Data":"52c4bd9d8ead3b9998e43d438b4c23b6e64b840a9fc2db3df3cbf294efc90911"} Jul 15 08:48:13.620039 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.620020 2567 scope.go:117] "RemoveContainer" containerID="50f2f195d3e1a6d3a16e6a143c8aa3608af56efc7182706b8129841c44ec2ab4" Jul 15 08:48:13.620117 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.620047 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7631-139826930779456" Jul 15 08:48:13.735718 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:13.735680 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:16.637682 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:16.637652 2567 generic.go:356] "Generic (PLEG): container finished" podID="b147395f-8227-44aa-b43b-d6fef3bbcd99" containerID="90c51f8f3dcbd52388ec95b2676cbaa5c0faa5ddf43c3dd83317ad4b10e99ee7" exitCode=0 Jul 15 08:48:16.638010 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:16.637721 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" event={"ID":"b147395f-8227-44aa-b43b-d6fef3bbcd99","Type":"ContainerDied","Data":"90c51f8f3dcbd52388ec95b2676cbaa5c0faa5ddf43c3dd83317ad4b10e99ee7"} Jul 15 08:48:17.772914 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:17.772890 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" Jul 15 08:48:17.845764 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:17.845737 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b147395f-8227-44aa-b43b-d6fef3bbcd99-maas-api-service-ca\") pod \"b147395f-8227-44aa-b43b-d6fef3bbcd99\" (UID: \"b147395f-8227-44aa-b43b-d6fef3bbcd99\") " Jul 15 08:48:17.846094 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:17.846075 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b147395f-8227-44aa-b43b-d6fef3bbcd99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b147395f-8227-44aa-b43b-d6fef3bbcd99" (UID: "b147395f-8227-44aa-b43b-d6fef3bbcd99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:48:17.946645 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:17.946583 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b147395f-8227-44aa-b43b-d6fef3bbcd99-maas-api-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:18.649221 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:18.649195 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" Jul 15 08:48:18.649398 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:18.649238 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj" event={"ID":"b147395f-8227-44aa-b43b-d6fef3bbcd99","Type":"ContainerDied","Data":"52c4bd9d8ead3b9998e43d438b4c23b6e64b840a9fc2db3df3cbf294efc90911"} Jul 15 08:48:18.649398 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:18.649289 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c4bd9d8ead3b9998e43d438b4c23b6e64b840a9fc2db3df3cbf294efc90911" Jul 15 08:48:23.496780 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:23.496747 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:27.142962 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:27.142932 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:48:28.086220 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:28.086186 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:34.816421 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:34.816389 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:34.885781 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:34.885723 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:38.858504 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:38.858464 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:39.167703 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.167615 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:39.252253 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.252201 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97"] Jul 15 08:48:39.252575 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.252562 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b147395f-8227-44aa-b43b-d6fef3bbcd99" containerName="revoke-keys" Jul 15 08:48:39.252632 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.252576 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b147395f-8227-44aa-b43b-d6fef3bbcd99" containerName="revoke-keys" Jul 15 08:48:39.252632 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.252629 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b147395f-8227-44aa-b43b-d6fef3bbcd99" containerName="revoke-keys" Jul 15 08:48:39.257486 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.257468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" Jul 15 08:48:39.260292 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.260273 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:48:39.260386 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.260273 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 08:48:39.265277 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.264984 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97"] Jul 15 08:48:39.403527 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.403498 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eced1c37-cc84-4728-bf64-971923834d16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1db148-kln97\" (UID: \"eced1c37-cc84-4728-bf64-971923834d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" Jul 15 08:48:39.503948 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.503872 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eced1c37-cc84-4728-bf64-971923834d16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1db148-kln97\" (UID: \"eced1c37-cc84-4728-bf64-971923834d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" Jul 15 08:48:39.504460 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.504428 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eced1c37-cc84-4728-bf64-971923834d16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1db148-kln97\" (UID: \"eced1c37-cc84-4728-bf64-971923834d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" Jul 15 08:48:39.569748 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.569728 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" Jul 15 08:48:39.701111 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.701085 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97"] Jul 15 08:48:39.703387 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:48:39.703356 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeced1c37_cc84_4728_bf64_971923834d16.slice/crio-6d1aa459bdcc465ea7a730947464296e29195a39297908775164023efb676b20 WatchSource:0}: Error finding container 6d1aa459bdcc465ea7a730947464296e29195a39297908775164023efb676b20: Status 404 returned error can't find the container with id 6d1aa459bdcc465ea7a730947464296e29195a39297908775164023efb676b20 Jul 15 08:48:39.763600 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.763541 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" event={"ID":"eced1c37-cc84-4728-bf64-971923834d16","Type":"ContainerStarted","Data":"6d1aa459bdcc465ea7a730947464296e29195a39297908775164023efb676b20"} Jul 15 08:48:39.854917 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.854895 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:39.955694 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:39.955673 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:40.770128 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:40.770044 2567 generic.go:356] "Generic (PLEG): container finished" podID="eced1c37-cc84-4728-bf64-971923834d16" containerID="1291590dbe1060cb7ef6819d406735206a696322aa9c090bd1e7fc82683c60cb" exitCode=0 Jul 15 08:48:40.770268 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:40.770132 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" event={"ID":"eced1c37-cc84-4728-bf64-971923834d16","Type":"ContainerDied","Data":"1291590dbe1060cb7ef6819d406735206a696322aa9c090bd1e7fc82683c60cb"} Jul 15 08:48:41.915912 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:41.915885 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" Jul 15 08:48:42.025101 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:42.025075 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eced1c37-cc84-4728-bf64-971923834d16-maas-api-service-ca\") pod \"eced1c37-cc84-4728-bf64-971923834d16\" (UID: \"eced1c37-cc84-4728-bf64-971923834d16\") " Jul 15 08:48:42.025487 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:42.025403 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eced1c37-cc84-4728-bf64-971923834d16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eced1c37-cc84-4728-bf64-971923834d16" (UID: "eced1c37-cc84-4728-bf64-971923834d16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:48:42.126207 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:42.126181 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eced1c37-cc84-4728-bf64-971923834d16-maas-api-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\"" Jul 15 08:48:42.781355 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:42.781328 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" Jul 15 08:48:42.781556 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:42.781322 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1db148-kln97" event={"ID":"eced1c37-cc84-4728-bf64-971923834d16","Type":"ContainerDied","Data":"6d1aa459bdcc465ea7a730947464296e29195a39297908775164023efb676b20"} Jul 15 08:48:42.781556 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:42.781474 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d1aa459bdcc465ea7a730947464296e29195a39297908775164023efb676b20" Jul 15 08:48:49.788498 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:49.788455 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:48:53.136948 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:53.136916 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fq8cd"] Jul 15 08:48:54.339932 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:48:54.339896 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:49:00.483181 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:00.483150 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:49:00.581385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:00.581358 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:49:05.071835 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:05.071798 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:49:05.328871 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:05.328795 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:49:11.995450 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:11.995417 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b8c85cff6-wmk8s_bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10/copy-manifests/0.log" Jul 15 08:49:12.009152 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:12.009127 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b8c85cff6-wmk8s_bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10/manager/0.log" Jul 15 08:49:14.077646 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:14.077612 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-8e59d0-2hhlj_b147395f-8227-44aa-b43b-d6fef3bbcd99/revoke-keys/0.log" Jul 15 08:49:14.198292 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:14.198264 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-1db148-kln97_eced1c37-cc84-4728-bf64-971923834d16/revoke-keys/0.log" Jul 15 08:49:15.166942 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.166911 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl_a1150c90-f7bf-430c-bebd-90f012094eda/util/0.log" Jul 15 08:49:15.186688 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.186661 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl_a1150c90-f7bf-430c-bebd-90f012094eda/pull/0.log" Jul 15 08:49:15.194292 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.194252 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl_a1150c90-f7bf-430c-bebd-90f012094eda/extract/0.log" Jul 15 08:49:15.316876 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.316833 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb_488a3ba2-7ee4-47dc-9238-352d8cd763bb/extract/0.log" Jul 15 08:49:15.323694 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.323674 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb_488a3ba2-7ee4-47dc-9238-352d8cd763bb/util/0.log" Jul 15 08:49:15.330384 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.330366 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb_488a3ba2-7ee4-47dc-9238-352d8cd763bb/pull/0.log" Jul 15 08:49:15.452480 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.452357 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8_085f1648-bcc8-4eea-9071-4dd0d7c35d44/pull/0.log" Jul 15 08:49:15.459111 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.459086 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8_085f1648-bcc8-4eea-9071-4dd0d7c35d44/extract/0.log" Jul 15 08:49:15.465258 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.465242 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8_085f1648-bcc8-4eea-9071-4dd0d7c35d44/util/0.log" Jul 15 08:49:15.585775 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.585749 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt_ba248e4a-2f01-47fe-b48e-4c241b1480d7/util/0.log" Jul 15 08:49:15.593526 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.593506 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt_ba248e4a-2f01-47fe-b48e-4c241b1480d7/pull/0.log" Jul 15 08:49:15.601765 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:15.601708 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt_ba248e4a-2f01-47fe-b48e-4c241b1480d7/extract/0.log" Jul 15 08:49:16.098090 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:16.098065 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-668lz_e24e7682-c745-4d46-8d14-858091923416/kuadrant-console-plugin/0.log" Jul 15 08:49:16.501796 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:16.501699 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fq8cd_6e010c21-46b3-4512-9c6a-d2e1a49d5956/limitador/0.log" Jul 15 08:49:17.391501 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:17.391466 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-657b79bd8d-xltvf_c011c351-7f03-49f2-a9d5-83fd4958f78d/kube-auth-proxy/0.log" Jul 15 08:49:18.728908 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:18.728861 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-rs758_40b52868-5c08-4866-8d73-81c9ad2b70bf/main/0.log" Jul 15 08:49:18.849874 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:18.849848 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-z7k2k_c4a43e78-af11-445c-83aa-c3dc3efc0e17/main/0.log" Jul 15 08:49:23.097944 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.097909 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:49:23.131292 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.131259 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bt6dg/must-gather-2hl74"] Jul 15 08:49:23.131641 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.131624 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eced1c37-cc84-4728-bf64-971923834d16" containerName="revoke-keys" Jul 15 08:49:23.131729 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.131643 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eced1c37-cc84-4728-bf64-971923834d16" containerName="revoke-keys" Jul 15 08:49:23.131785 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.131737 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eced1c37-cc84-4728-bf64-971923834d16" containerName="revoke-keys" Jul 15 08:49:23.134723 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.134704 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bt6dg/must-gather-2hl74" Jul 15 08:49:23.137167 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.137141 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bt6dg\"/\"openshift-service-ca.crt\"" Jul 15 08:49:23.137300 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.137278 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bt6dg\"/\"kube-root-ca.crt\"" Jul 15 08:49:23.138013 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.138000 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bt6dg\"/\"default-dockercfg-nbpnw\"" Jul 15 08:49:23.145213 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.145191 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bt6dg/must-gather-2hl74"] Jul 15 08:49:23.218760 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.218696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrp5x\" (UniqueName: \"kubernetes.io/projected/159f0764-158d-443e-b796-f7041f047d9d-kube-api-access-qrp5x\") pod \"must-gather-2hl74\" (UID: \"159f0764-158d-443e-b796-f7041f047d9d\") " pod="openshift-must-gather-bt6dg/must-gather-2hl74" Jul 15 08:49:23.218760 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.218744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/159f0764-158d-443e-b796-f7041f047d9d-must-gather-output\") pod \"must-gather-2hl74\" (UID: \"159f0764-158d-443e-b796-f7041f047d9d\") " pod="openshift-must-gather-bt6dg/must-gather-2hl74" Jul 15 08:49:23.319878 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.319845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/159f0764-158d-443e-b796-f7041f047d9d-must-gather-output\") pod \"must-gather-2hl74\" (UID: \"159f0764-158d-443e-b796-f7041f047d9d\") " pod="openshift-must-gather-bt6dg/must-gather-2hl74" Jul 15 08:49:23.319985 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.319914 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrp5x\" (UniqueName: \"kubernetes.io/projected/159f0764-158d-443e-b796-f7041f047d9d-kube-api-access-qrp5x\") pod \"must-gather-2hl74\" (UID: \"159f0764-158d-443e-b796-f7041f047d9d\") " pod="openshift-must-gather-bt6dg/must-gather-2hl74" Jul 15 08:49:23.320160 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.320144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/159f0764-158d-443e-b796-f7041f047d9d-must-gather-output\") pod \"must-gather-2hl74\" (UID: \"159f0764-158d-443e-b796-f7041f047d9d\") " pod="openshift-must-gather-bt6dg/must-gather-2hl74" Jul 15 08:49:23.330270 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.330249 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrp5x\" (UniqueName: \"kubernetes.io/projected/159f0764-158d-443e-b796-f7041f047d9d-kube-api-access-qrp5x\") pod \"must-gather-2hl74\" (UID: \"159f0764-158d-443e-b796-f7041f047d9d\") " pod="openshift-must-gather-bt6dg/must-gather-2hl74" Jul 15 08:49:23.444575 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.444554 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bt6dg/must-gather-2hl74" Jul 15 08:49:23.573392 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.573367 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bt6dg/must-gather-2hl74"] Jul 15 08:49:23.575753 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:49:23.575719 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod159f0764_158d_443e_b796_f7041f047d9d.slice/crio-7fcfbbfc11697f43e1dff52f34fb0b30f48a4a2cb7c593d362184b9cba4172e3 WatchSource:0}: Error finding container 7fcfbbfc11697f43e1dff52f34fb0b30f48a4a2cb7c593d362184b9cba4172e3: Status 404 returned error can't find the container with id 7fcfbbfc11697f43e1dff52f34fb0b30f48a4a2cb7c593d362184b9cba4172e3 Jul 15 08:49:23.989639 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:23.989602 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bt6dg/must-gather-2hl74" event={"ID":"159f0764-158d-443e-b796-f7041f047d9d","Type":"ContainerStarted","Data":"7fcfbbfc11697f43e1dff52f34fb0b30f48a4a2cb7c593d362184b9cba4172e3"} Jul 15 08:49:24.350690 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:24.350646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" limit=1024 Jul 15 08:49:24.999213 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:24.999114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bt6dg/must-gather-2hl74" event={"ID":"159f0764-158d-443e-b796-f7041f047d9d","Type":"ContainerStarted","Data":"753e84007a38f3df735e952afa498580fdefb5e7f1df0c51a62194a42f7193b7"} Jul 15 08:49:24.999213 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:24.999169 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bt6dg/must-gather-2hl74" event={"ID":"159f0764-158d-443e-b796-f7041f047d9d","Type":"ContainerStarted","Data":"906712b0cba254e416aad3d63a39c932f898d12efd5074bb933fabad2c3724a0"} Jul 15 08:49:26.124303 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:26.124270 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nlzkz_e8a42fee-4675-42df-8f1a-71cb407b62ad/global-pull-secret-syncer/0.log" Jul 15 08:49:26.181704 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:26.181647 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fz4s9_c517d898-f089-47b7-938d-7f6c2d0339cf/konnectivity-agent/0.log" Jul 15 08:49:26.286759 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:26.286731 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-37.ec2.internal_9545b0e517d36e2a7fb3813f2cac8af0/haproxy/0.log" Jul 15 08:49:30.292838 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.292790 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl_a1150c90-f7bf-430c-bebd-90f012094eda/extract/0.log" Jul 15 08:49:30.325925 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.325842 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl_a1150c90-f7bf-430c-bebd-90f012094eda/util/0.log" Jul 15 08:49:30.351149 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.351117 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzlvl_a1150c90-f7bf-430c-bebd-90f012094eda/pull/0.log" Jul 15 08:49:30.383205 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.383121 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb_488a3ba2-7ee4-47dc-9238-352d8cd763bb/extract/0.log" Jul 15 08:49:30.410030 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.410008 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb_488a3ba2-7ee4-47dc-9238-352d8cd763bb/util/0.log" Jul 15 08:49:30.447238 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.447194 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwvxb_488a3ba2-7ee4-47dc-9238-352d8cd763bb/pull/0.log" Jul 15 08:49:30.482620 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.482587 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8_085f1648-bcc8-4eea-9071-4dd0d7c35d44/extract/0.log" Jul 15 08:49:30.506002 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.505972 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8_085f1648-bcc8-4eea-9071-4dd0d7c35d44/util/0.log" Jul 15 08:49:30.528454 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.528410 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mdgt8_085f1648-bcc8-4eea-9071-4dd0d7c35d44/pull/0.log" Jul 15 08:49:30.562116 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.561975 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt_ba248e4a-2f01-47fe-b48e-4c241b1480d7/extract/0.log" Jul 15 08:49:30.588048 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.588000 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt_ba248e4a-2f01-47fe-b48e-4c241b1480d7/util/0.log" Jul 15 08:49:30.621778 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:30.621734 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12vjpt_ba248e4a-2f01-47fe-b48e-4c241b1480d7/pull/0.log" Jul 15 08:49:31.287664 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:31.287640 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-668lz_e24e7682-c745-4d46-8d14-858091923416/kuadrant-console-plugin/0.log" Jul 15 08:49:31.531689 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:31.531660 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fq8cd_6e010c21-46b3-4512-9c6a-d2e1a49d5956/limitador/0.log" Jul 15 08:49:33.706758 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:33.706724 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8rrq7_08366be9-0b75-46cf-b941-d181ee2c8b51/kube-state-metrics/0.log" Jul 15 08:49:33.729602 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:33.729573 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8rrq7_08366be9-0b75-46cf-b941-d181ee2c8b51/kube-rbac-proxy-main/0.log" Jul 15 08:49:33.754772 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:33.754699 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8rrq7_08366be9-0b75-46cf-b941-d181ee2c8b51/kube-rbac-proxy-self/0.log" Jul 15 08:49:33.789813 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:33.789785 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6567f9cb8b-lgvbl_069b1964-cf1e-457f-bc61-7b9ba4b383da/metrics-server/0.log" Jul 15 08:49:33.852515 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:33.852489 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rs4n_e15d3bdc-32c7-4041-9d8c-c38459b2a1a1/node-exporter/0.log" Jul 15 08:49:33.879214 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:33.879154 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rs4n_e15d3bdc-32c7-4041-9d8c-c38459b2a1a1/kube-rbac-proxy/0.log" Jul 15 08:49:33.913419 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:33.913376 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rs4n_e15d3bdc-32c7-4041-9d8c-c38459b2a1a1/init-textfile/0.log" Jul 15 08:49:34.106809 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:34.106784 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-ndxf4_a1a83d63-7ef9-4d16-be23-cc1375e6822a/kube-rbac-proxy-main/0.log" Jul 15 08:49:34.131923 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:34.131898 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-ndxf4_a1a83d63-7ef9-4d16-be23-cc1375e6822a/kube-rbac-proxy-self/0.log" Jul 15 08:49:34.154060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:34.154033 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-ndxf4_a1a83d63-7ef9-4d16-be23-cc1375e6822a/openshift-state-metrics/0.log" Jul 15 08:49:35.584496 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.584419 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bt6dg/must-gather-2hl74" podStartSLOduration=11.731248594 podStartE2EDuration="12.584396092s" totalImagesPullingTime="853.147498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:49:25.014643301 +0000 UTC m=+2922.530440136" watchObservedRunningTime="2026-07-15 08:49:35.584396092 +0000 UTC m=+2933.100192927" Jul 15 08:49:35.588457 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.588413 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt"] Jul 15 08:49:35.593976 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.593954 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.599628 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.599604 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt"] Jul 15 08:49:35.634885 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.634852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-sys\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.635176 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.635151 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-lib-modules\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.635385 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.635372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-proc\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.635535 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.635521 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2d5x\" (UniqueName: \"kubernetes.io/projected/f3453917-7e4d-4fce-939c-c600593e2782-kube-api-access-g2d5x\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.635690 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.635674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-podres\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.736664 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.736625 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-podres\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.736851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.736675 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-sys\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.736851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.736755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-sys\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.736851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.736799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-lib-modules\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.736851 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.736823 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-podres\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.737092 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.736853 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-proc\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.737092 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.736874 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2d5x\" (UniqueName: \"kubernetes.io/projected/f3453917-7e4d-4fce-939c-c600593e2782-kube-api-access-g2d5x\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.737261 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.737112 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-lib-modules\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.737386 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.737362 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3453917-7e4d-4fce-939c-c600593e2782-proc\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.747845 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.747816 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2d5x\" (UniqueName: \"kubernetes.io/projected/f3453917-7e4d-4fce-939c-c600593e2782-kube-api-access-g2d5x\") pod \"perf-node-gather-daemonset-wdbnt\" (UID: \"f3453917-7e4d-4fce-939c-c600593e2782\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:35.912970 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:35.912932 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:36.098918 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:36.098820 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt"] Jul 15 08:49:36.103853 ip-10-0-128-37 kubenswrapper[2567]: W0715 08:49:36.103814 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3453917_7e4d_4fce_939c_c600593e2782.slice/crio-413b5866e20cb74f71a2d048d16e4f42ae34e2ba47f480cc5279d9196f0057b6 WatchSource:0}: Error finding container 413b5866e20cb74f71a2d048d16e4f42ae34e2ba47f480cc5279d9196f0057b6: Status 404 returned error can't find the container with id 413b5866e20cb74f71a2d048d16e4f42ae34e2ba47f480cc5279d9196f0057b6 Jul 15 08:49:37.056805 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:37.056774 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-2274l_fd0a15dd-889a-4c64-a1bb-6f41a6fa9161/download-server/0.log" Jul 15 08:49:37.091871 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:37.091831 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" event={"ID":"f3453917-7e4d-4fce-939c-c600593e2782","Type":"ContainerStarted","Data":"72f1958dc2f726872c29ba5bdd10385f8716162729a5b5729abb583cf0af5749"} Jul 15 08:49:37.092060 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:37.091903 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" event={"ID":"f3453917-7e4d-4fce-939c-c600593e2782","Type":"ContainerStarted","Data":"413b5866e20cb74f71a2d048d16e4f42ae34e2ba47f480cc5279d9196f0057b6"} Jul 15 08:49:37.092139 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:37.092084 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:38.750425 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:38.750400 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m8h8b_00bae080-60c7-49c2-aff2-93e41d0e23be/dns/0.log" Jul 15 08:49:38.773651 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:38.773618 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m8h8b_00bae080-60c7-49c2-aff2-93e41d0e23be/kube-rbac-proxy/0.log" Jul 15 08:49:38.875664 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:38.875638 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-txzrm_7e5df8c5-cdfe-4f96-b7a9-50f5a4bad2c2/dns-node-resolver/0.log" Jul 15 08:49:39.488945 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:39.488914 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f4b9f5896-h7vrj_a7779880-a9eb-4e40-8499-e65c92b6034d/registry/0.log" Jul 15 08:49:39.511725 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:39.511696 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-clz7b_451a8c3d-179c-4a38-a3c3-50c96043ebf1/node-ca/0.log" Jul 15 08:49:40.695660 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:40.695622 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-657b79bd8d-xltvf_c011c351-7f03-49f2-a9d5-83fd4958f78d/kube-auth-proxy/0.log" Jul 15 08:49:41.655227 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:41.655199 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7gwv_b07d8157-8401-4516-ac0d-20c2d5c65275/serve-healthcheck-canary/0.log" Jul 15 08:49:42.226142 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:42.226088 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lnlh8_18bef297-7e7a-40d4-ac55-291487d73ee7/kube-rbac-proxy/0.log" Jul 15 08:49:42.247530 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:42.247482 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lnlh8_18bef297-7e7a-40d4-ac55-291487d73ee7/exporter/0.log" Jul 15 08:49:42.269993 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:42.269969 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lnlh8_18bef297-7e7a-40d4-ac55-291487d73ee7/extractor/0.log" Jul 15 08:49:43.109460 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:43.109261 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" Jul 15 08:49:43.131639 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:43.131592 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-wdbnt" podStartSLOduration=8.131575129 podStartE2EDuration="8.131575129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:49:37.109324701 +0000 UTC m=+2934.625121539" watchObservedRunningTime="2026-07-15 08:49:43.131575129 +0000 UTC m=+2940.647371960" Jul 15 08:49:45.302358 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:45.302279 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b8c85cff6-wmk8s_bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10/manager/0.log" Jul 15 08:49:45.324338 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:45.324311 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b8c85cff6-wmk8s_bbaf33b4-60e3-47c8-b5e8-2a6f44aa1a10/copy-manifests/0.log" Jul 15 08:49:46.525944 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:46.525910 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bdbd9fc98-722zt_d924e4e8-08db-4342-bf5a-efeea9fe23bb/manager/0.log" Jul 15 08:49:46.801307 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:46.801232 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-657b79bd8d-xltvf_c011c351-7f03-49f2-a9d5-83fd4958f78d/kube-auth-proxy/0.log" Jul 15 08:49:48.247710 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:48.247676 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-657b79bd8d-xltvf_c011c351-7f03-49f2-a9d5-83fd4958f78d/kube-auth-proxy/0.log" Jul 15 08:49:51.673673 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:51.673640 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-mqshx_eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37/migrator/0.log" Jul 15 08:49:51.693992 ip-10-0-128-37 kubenswrapper[2567]: I0715 08:49:51.693968 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-mqshx_eb3ce00e-fb28-4ef5-bdd2-46ac8e781c37/graceful-termination/0.log"