Jul 16 21:37:07.146022 ip-10-0-133-93 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 21:37:07.146037 ip-10-0-133-93 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 21:37:07.146047 ip-10-0-133-93 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 21:37:07.146555 ip-10-0-133-93 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 21:37:17.202729 ip-10-0-133-93 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 21:37:17.202747 ip-10-0-133-93 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 89aa2f96e0e241eab442f37d0793b4d1 -- Jul 16 21:39:12.845301 ip-10-0-133-93 systemd[1]: Starting Kubernetes Kubelet... Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.220228 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226733 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226752 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226756 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226760 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226763 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226766 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226769 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:39:13.263702 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226774 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226778 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226781 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226783 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226786 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226788 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226792 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226794 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226797 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226799 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226802 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226808 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226812 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226815 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226818 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226821 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226824 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226827 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226829 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226832 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:39:13.265127 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226834 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226837 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226839 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226842 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226845 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226848 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226850 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226853 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226856 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226859 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226862 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226865 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226867 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226870 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226873 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226875 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226878 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226880 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226883 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226885 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:39:13.265781 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226888 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226890 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226893 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226896 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226898 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226902 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226905 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226908 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226910 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226913 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226915 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226918 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226920 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226923 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226925 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226928 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226931 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226934 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226937 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226939 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:39:13.267057 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226943 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226947 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226950 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226953 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226956 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226959 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226961 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226964 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226967 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226970 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226973 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226977 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226979 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226982 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226985 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226988 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226990 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226993 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226996 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.226998 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:39:13.267704 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227001 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227004 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227006 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227010 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227014 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227016 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227019 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227023 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227025 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227028 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227031 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227033 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227775 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227783 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227786 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227790 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227793 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227796 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227799 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:39:13.268461 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227802 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227804 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227807 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227809 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227813 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227816 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227819 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227822 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227825 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227827 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227830 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227833 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227835 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227838 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227840 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227843 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227846 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227851 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227854 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:39:13.269029 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227857 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227860 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227863 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227865 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227868 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227871 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227874 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227876 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227878 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227881 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227883 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227887 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227889 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227892 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227895 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227905 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227910 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227913 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227916 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227919 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:39:13.269669 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227922 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227924 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227927 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227930 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227932 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227935 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227937 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227940 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227942 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227945 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227947 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227950 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227952 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227955 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227958 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227961 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227963 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227966 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227968 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227971 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:39:13.270358 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227974 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227976 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227979 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227981 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227987 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227990 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227993 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227996 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.227999 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228002 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228004 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228007 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228009 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228012 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228014 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228016 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228019 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228022 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228025 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:39:13.271209 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228028 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228031 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228034 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228036 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228039 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228041 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228044 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228046 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228048 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228051 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228055 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228059 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228061 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228064 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228472 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228477 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228481 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228484 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228487 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:39:13.271982 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228490 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228494 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228497 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228500 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228502 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228505 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228507 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228510 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228513 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228533 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228536 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228539 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228542 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228544 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228547 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228549 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228552 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228555 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228557 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228560 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:39:13.272676 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228563 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228565 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228568 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228570 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228573 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228576 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228579 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228582 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228584 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228587 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228589 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228592 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228595 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228599 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228602 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228604 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228607 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228610 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228613 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:39:13.273260 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228616 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228618 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228621 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228623 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228627 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228631 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228633 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228636 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228639 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228642 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228645 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228647 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228650 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228653 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228656 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228659 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228661 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228664 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228668 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:39:13.273912 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228672 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228675 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228679 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228682 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228685 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228687 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228690 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228695 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228697 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228700 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228702 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228706 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228709 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228712 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228714 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228717 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228721 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228725 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228729 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228732 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:39:13.274812 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228735 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228738 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228741 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228743 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228746 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228749 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228752 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228754 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228757 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228760 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228763 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228765 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228768 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228771 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228773 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.228776 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228855 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228867 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228873 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228880 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228885 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 21:39:13.346464 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228888 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 21:39:13.296466 ip-10-0-133-93 systemd[1]: Started Kubernetes Kubelet. Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228892 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228896 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228899 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228903 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228906 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228909 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228913 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228916 2573 flags.go:64] FLAG: --cgroup-root="" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228918 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228922 2573 flags.go:64] FLAG: --client-ca-file="" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228924 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228927 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228932 2573 flags.go:64] FLAG: --cluster-domain="" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228935 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228938 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228941 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228945 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228950 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228954 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228957 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228960 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228964 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228967 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 21:39:13.347898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228970 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228973 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228976 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228980 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228983 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228986 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228990 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228994 2573 flags.go:64] FLAG: --enable-server="true" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.228997 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229001 2573 flags.go:64] FLAG: --event-burst="100" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229004 2573 flags.go:64] FLAG: --event-qps="50" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229007 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229010 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229013 2573 flags.go:64] FLAG: --eviction-hard="" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229017 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229020 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229023 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229026 2573 flags.go:64] FLAG: --eviction-soft="" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229029 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229031 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229034 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229037 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229040 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229043 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229046 2573 flags.go:64] FLAG: --feature-gates="" Jul 16 21:39:13.349548 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229050 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229054 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229057 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229060 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229064 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229067 2573 flags.go:64] FLAG: --help="false" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229069 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229072 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229075 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229078 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229082 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229085 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229088 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229092 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229095 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229099 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229102 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229105 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229108 2573 flags.go:64] FLAG: --kube-reserved="" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229111 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229113 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229117 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229119 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229122 2573 flags.go:64] FLAG: --lock-file="" Jul 16 21:39:13.350703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229125 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229128 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229131 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229137 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229139 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229142 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229145 2573 flags.go:64] FLAG: --logging-format="text" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229147 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229151 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229155 2573 flags.go:64] FLAG: --manifest-url="" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229158 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229164 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229168 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229172 2573 flags.go:64] FLAG: --max-pods="110" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229175 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229178 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229181 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229184 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229187 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229190 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229193 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229203 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229206 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229210 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 21:39:13.352492 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229213 2573 flags.go:64] FLAG: --pod-cidr="" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229217 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229223 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229226 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229229 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229231 2573 flags.go:64] FLAG: --port="10250" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229234 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229237 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-039ba3f10fe391d2c" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229240 2573 flags.go:64] FLAG: --qos-reserved="" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229243 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229246 2573 flags.go:64] FLAG: --register-node="true" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229249 2573 flags.go:64] FLAG: --register-with-taints="" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229252 2573 flags.go:64] FLAG: --registry-burst="10" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229255 2573 flags.go:64] FLAG: --registry-qps="5" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229258 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229261 2573 flags.go:64] FLAG: --reserved-memory="" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229264 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229267 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229272 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229275 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229278 2573 flags.go:64] FLAG: --runonce="false" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229281 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229284 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229286 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229289 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 21:39:13.353373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229292 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229295 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229298 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229301 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229305 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229308 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229310 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229313 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229317 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229320 2573 flags.go:64] FLAG: --system-cgroups="" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229323 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229328 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229331 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229334 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229340 2573 flags.go:64] FLAG: --tls-min-version="" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229343 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229346 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229349 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229352 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229355 2573 flags.go:64] FLAG: --v="2" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229359 2573 flags.go:64] FLAG: --version="false" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229363 2573 flags.go:64] FLAG: --vmodule="" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229367 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229371 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229486 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:39:13.354427 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229492 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229495 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229498 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229500 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229503 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229506 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229508 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229511 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229514 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229535 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229539 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229543 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229546 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229548 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229551 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229554 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229557 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229560 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229563 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229565 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:39:13.355351 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229568 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229570 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229573 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229575 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229579 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229583 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229586 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229589 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229592 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229595 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229598 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229601 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229605 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229608 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229610 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229613 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229615 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229618 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229621 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:39:13.356257 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229623 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229626 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229629 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229631 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229636 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229640 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229643 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229646 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229648 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229651 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229654 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229657 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229660 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229662 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229665 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229667 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229670 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229674 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229677 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229680 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:39:13.356946 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229683 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229686 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229689 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229691 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229693 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229698 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229700 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229703 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229706 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229709 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229712 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229714 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229717 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229719 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229722 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229725 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229728 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229731 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229734 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229737 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:39:13.357879 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229740 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229742 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229745 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229748 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229750 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229753 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229755 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229758 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229761 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229763 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229766 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229768 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229771 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229773 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229776 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229778 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229781 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229785 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:39:13.358591 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.229787 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.229793 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.238417 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.238439 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.239196 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.239204 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239270 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239276 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239280 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239284 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239287 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239291 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239294 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239296 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:39:13.359432 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239299 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239302 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239305 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239307 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239310 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239313 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239316 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239318 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239321 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239324 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239326 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239330 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239332 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239335 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239338 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239340 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239343 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239346 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239349 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239351 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:39:13.359964 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239355 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239358 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239361 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239363 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239366 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239368 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239371 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239373 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239376 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239378 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239381 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239383 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239385 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239388 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239391 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239394 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239397 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239399 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239403 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:39:13.360985 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239407 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239410 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239417 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239420 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239423 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239426 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239429 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239431 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239434 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239438 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239441 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239444 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239447 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239450 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239453 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239455 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239458 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239460 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239463 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239466 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:39:13.460355 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239468 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239471 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239474 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239476 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239479 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239481 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239484 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239486 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239488 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239491 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239494 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239496 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239500 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239503 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239508 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239512 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239515 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239527 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239530 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:39:13.478627 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239533 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239535 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239538 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239543 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239545 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239548 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239550 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239553 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239555 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239558 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239560 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239563 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239566 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.239572 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239697 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:39:13.479532 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239703 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239706 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239709 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239713 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239718 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239720 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239723 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239726 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239729 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239731 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239734 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239738 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239741 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239746 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239750 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239754 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239757 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239760 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239763 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:39:13.481530 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239766 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239770 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239773 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239776 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239779 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239782 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239785 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239787 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239790 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239792 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239795 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239798 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239800 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239803 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239805 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239808 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239810 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239813 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239816 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239818 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:39:13.482211 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239821 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239823 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239826 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239828 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239831 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239834 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239838 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239841 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239843 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239846 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239848 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239851 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239853 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239857 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239860 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239863 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239865 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239868 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239871 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239873 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:39:13.483515 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239876 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239878 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239881 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239883 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239886 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239889 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239891 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239894 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239896 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239899 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239901 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239904 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239906 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239908 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239911 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239913 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239916 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239918 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239924 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:39:13.484175 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239926 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239929 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239931 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239934 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239936 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239939 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239943 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239946 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239948 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239951 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239954 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239956 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239959 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239963 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239966 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239969 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239973 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239976 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239978 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:39:13.485112 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:13.239981 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.239986 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.240116 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.242357 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.243846 2573 server.go:1019] "Starting client certificate rotation" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.243950 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.243984 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.246408 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.247682 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.256432 2573 log.go:25] "Validated CRI v1 runtime API" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.258425 2573 log.go:25] "Validated CRI v1 image API" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.262574 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 21:39:13.485749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.268079 2573 fs.go:135] Filesystem UUIDs: map[07b6a94d-a477-4b8f-82ec-49dd89732ef3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a74c0150-43b1-45e7-93a7-9534e9d0d7bc:/dev/nvme0n1p3] Jul 16 21:39:13.486477 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.268096 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 21:39:13.486640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.273878 2573 manager.go:216] Machine: {Timestamp:2026-07-16 21:39:13.27334315 +0000 UTC m=+0.315567380 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110180 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21cadf8aa05efd0a5aacc250b00232 SystemUUID:ec21cadf-8aa0-5efd-0a5a-acc250b00232 BootID:89aa2f96-e0e2-41ea-b442-f37d0793b4d1 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9b:e5:8b:8b:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9b:e5:8b:8b:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:44:5b:b4:32:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 21:39:13.486640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.273984 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 21:39:13.486640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.274070 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 21:39:13.486640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.275719 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 21:39:13.486640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.275758 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-93.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 21:39:13.486640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.276498 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 21:39:13.486640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.276511 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 21:39:13.486640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.276532 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.276549 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.276665 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.277665 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.280545 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.280673 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.280701 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.280712 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.280738 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.280747 2573 kubelet.go:398] "Adding apiserver pod source" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.280762 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.281086 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.281101 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.284026 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.285916 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.285949 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286900 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286913 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286920 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286926 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286935 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286940 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286949 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286955 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286962 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286968 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286976 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.286983 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.288892 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.288903 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.294275 2573 server.go:1268] "Started kubelet" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.294328 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-93.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.294364 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.294413 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.294422 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.294540 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.294574 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.297657 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.298679 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 16 21:39:13.487308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.299787 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lq2hn" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.300145 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.299711 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-93.ec2.internal.18c2e2f718df6233 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-93.ec2.internal,UID:ip-10-0-133-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-93.ec2.internal,},FirstTimestamp:2026-07-16 21:39:13.294230067 +0000 UTC m=+0.336454295,LastTimestamp:2026-07-16 21:39:13.294230067 +0000 UTC m=+0.336454295,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-93.ec2.internal,}" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.302642 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.302686 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.303386 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.303397 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.305134 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.305196 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.305201 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.306045 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.306692 2573 factory.go:55] Registering systemd factory Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.306705 2573 factory.go:223] Registration of the systemd container factory successfully Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.306870 2573 factory.go:153] Registering CRI-O factory Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.306876 2573 factory.go:223] Registration of the crio container factory successfully Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.306918 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.306938 2573 factory.go:103] Registering Raw factory Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.306948 2573 manager.go:1196] Started watching for new ooms in manager Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.307285 2573 manager.go:349] Starting recovery of all containers Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.311756 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lq2hn" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.312400 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.312509 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.329938 2573 manager.go:354] Recovery completed Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.334998 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.337099 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.337124 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.337136 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.337928 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.337939 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.337957 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.339168 2573 policy_none.go:49] "None policy: Start" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.339182 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.339192 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.340236 2573 policy_none.go:47] "Start" Jul 16 21:39:13.488665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.340952 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.341904 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.341926 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.341936 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.384456 2573 manager.go:337] "Starting Device Plugin manager" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.384502 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.384515 2573 server.go:87] "Starting device plugin registration server" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.384866 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.384899 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.390586 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.390758 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.390768 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.391569 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.391630 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.485954 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.487109 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.487153 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.487172 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.487212 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.494441 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.494476 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-93.ec2.internal\": node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.509840 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.532845 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.534179 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.534222 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.534251 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.534288 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.536832 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:39:13.619531 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.610937 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.635957 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal"] Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.636041 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.637875 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.637905 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.637917 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.639258 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.639371 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.639400 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.640146 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.640168 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.640146 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.640178 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.640219 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.640253 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.641634 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.641657 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.642461 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.642487 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.642498 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.674288 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-93.ec2.internal\" not found" node="ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.676755 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-93.ec2.internal\" not found" node="ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.708272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a159e8edac5a31d9fd33bd8965a7119b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"a159e8edac5a31d9fd33bd8965a7119b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.708320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a159e8edac5a31d9fd33bd8965a7119b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"a159e8edac5a31d9fd33bd8965a7119b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.708346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad611d48b80ded02db93237655788291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-93.ec2.internal\" (UID: \"ad611d48b80ded02db93237655788291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.760124 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.711469 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:13.873567 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.809400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a159e8edac5a31d9fd33bd8965a7119b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"a159e8edac5a31d9fd33bd8965a7119b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.873567 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.809468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a159e8edac5a31d9fd33bd8965a7119b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"a159e8edac5a31d9fd33bd8965a7119b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.873567 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.809494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad611d48b80ded02db93237655788291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-93.ec2.internal\" (UID: \"ad611d48b80ded02db93237655788291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.873567 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.809587 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad611d48b80ded02db93237655788291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-93.ec2.internal\" (UID: \"ad611d48b80ded02db93237655788291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.873567 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.809651 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a159e8edac5a31d9fd33bd8965a7119b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"a159e8edac5a31d9fd33bd8965a7119b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.873567 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.809678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a159e8edac5a31d9fd33bd8965a7119b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"a159e8edac5a31d9fd33bd8965a7119b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.873567 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.811607 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:13.912243 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:13.912201 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:13.981721 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.981684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" Jul 16 21:39:13.981808 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:13.981727 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:14.012829 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.012611 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:14.113598 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.113552 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found" Jul 16 21:39:14.114267 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.114250 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:39:14.206638 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.206598 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" Jul 16 21:39:14.218772 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.218748 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:39:14.221652 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.221633 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:14.242849 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.242820 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 21:39:14.243017 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.242992 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 21:39:14.243066 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.243019 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 21:39:14.243066 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.243051 2573 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a617a8d19ff59485db3937b03626dadb-98f3f004a891bbcb.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.133.93:51740->3.90.192.247:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" Jul 16 21:39:14.282656 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.282564 2573 apiserver.go:52] "Watching apiserver" Jul 16 21:39:14.303189 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.303153 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 21:39:14.303976 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.303955 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 21:39:14.304355 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.304332 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-z4zlv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt","openshift-cluster-node-tuning-operator/tuned-wvcf7","openshift-multus/network-metrics-daemon-qhkrt","openshift-network-diagnostics/network-check-target-4qbsw","openshift-network-operator/iptables-alerter-4b4zv","openshift-ovn-kubernetes/ovnkube-node-zd2ck","kube-system/global-pull-secret-syncer-db75d","kube-system/konnectivity-agent-7ffsb","kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal","openshift-dns/node-resolver-5jktm","openshift-image-registry/node-ca-knpg7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal","openshift-multus/multus-additional-cni-plugins-zbvvv"] Jul 16 21:39:14.306066 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.306041 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.307397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.307378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.308675 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.308660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.309902 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.309886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:14.309979 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.309955 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:14.311046 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311029 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:14.311155 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.311093 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:14.311830 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-var-lib-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.311830 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.311989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-ovn\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.311989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311857 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-cni-netd\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.311989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-systemd\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.311989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311878 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ls9vj\"" Jul 16 21:39:14.311989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311909 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-log-socket\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.311989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.311999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-ovnkube-config\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312024 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfwrl\" (UniqueName: \"kubernetes.io/projected/9a30d648-865f-476d-99a2-40f26f72ab32-kube-api-access-nfwrl\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-systemd-units\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-node-log\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312074 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312081 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-cni-bin\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-env-overrides\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-ovnkube-script-lib\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-kubelet\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-run-netns\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-etc-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.312320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a30d648-865f-476d-99a2-40f26f72ab32-ovn-node-metrics-cert\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312354 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312366 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312345 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-slash\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312379 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312442 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312781 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312862 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312887 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wct5n\"" Jul 16 21:39:14.313149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.312863 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-78kmx\"" Jul 16 21:39:14.313645 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.313269 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 21:39:14.313645 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.313297 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 21:39:14.313843 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.313828 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:14.315255 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.315240 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.316727 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.316703 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 21:39:14.316866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.316771 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:39:14.316866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.316836 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.317004 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.316922 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8x7x5\"" Jul 16 21:39:14.317004 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.316993 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 21:39:14.317106 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.316996 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 21:39:14.317319 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317261 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:34:14 +0000 UTC" deadline="2028-03-12 10:37:37.056958619 +0000 UTC" Jul 16 21:39:14.317319 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317309 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 21:39:14.317501 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317311 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 21:39:14.317501 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317313 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14508h58m22.739650069s" Jul 16 21:39:14.317501 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317417 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hnks5\"" Jul 16 21:39:14.317994 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317776 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 21:39:14.317994 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317779 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 21:39:14.317994 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317822 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hlz8m\"" Jul 16 21:39:14.317994 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.317913 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 21:39:14.318229 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.318093 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 21:39:14.318229 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.318117 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.318229 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.318165 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 21:39:14.318229 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.318196 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:14.319733 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.319693 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.320244 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.320228 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 21:39:14.320360 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.320343 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:39:14.320951 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.320931 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 21:39:14.321275 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.321256 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.321275 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.321268 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t7bcb\"" Jul 16 21:39:14.323269 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.322888 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l682l\"" Jul 16 21:39:14.323269 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.322936 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 21:39:14.323269 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.323141 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 21:39:14.325619 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.325262 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 21:39:14.325619 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.325383 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 21:39:14.325619 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.325442 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5m5wl\"" Jul 16 21:39:14.326304 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.326286 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 21:39:14.349622 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.349591 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9l4d" Jul 16 21:39:14.361779 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.361752 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9l4d" Jul 16 21:39:14.406275 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.406251 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 21:39:14.412647 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-var-lib-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.412647 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-ovn\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-cni-netd\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-sys-fs\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-var-lib-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-tuned\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-tmp\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-ovnkube-config\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-ovn\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412779 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-cni-netd\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412774 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysconfig\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.412864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-system-cni-dir\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-system-cni-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-cni-binary-copy\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-cni-bin\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11b603d-097d-4bca-b187-43a143774e00-tmp-dir\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.412987 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmv4\" (UniqueName: \"kubernetes.io/projected/ffa68804-e8e5-473d-8bc2-b502e45313f1-kube-api-access-hsmv4\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-cni-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-run-netns\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-etc-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413110 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-etc-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-lib-modules\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413137 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-run-netns\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xz7\" (UniqueName: \"kubernetes.io/projected/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-kube-api-access-z7xz7\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.413328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413229 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-slash\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413264 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-slash\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c91f3cb3-36af-4a18-8e17-0da940e6ef85-serviceca\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-modprobe-d\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-var-lib-kubelet\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413400 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-openvswitch\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-log-socket\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfwrl\" (UniqueName: \"kubernetes.io/projected/9a30d648-865f-476d-99a2-40f26f72ab32-kube-api-access-nfwrl\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b11b603d-097d-4bca-b187-43a143774e00-hosts-file\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-log-socket\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-cni-bin\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.413942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-env-overrides\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghd7k\" (UniqueName: \"kubernetes.io/projected/c91f3cb3-36af-4a18-8e17-0da940e6ef85-kube-api-access-ghd7k\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413639 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-cni-bin\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9975j\" (UniqueName: \"kubernetes.io/projected/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-kube-api-access-9975j\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysctl-d\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-systemd\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-os-release\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4lx\" (UniqueName: \"kubernetes.io/projected/b11b603d-097d-4bca-b187-43a143774e00-kube-api-access-9c4lx\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-registration-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b07dc994-256a-44d4-8405-715a10a1feb6-konnectivity-ca\") pod \"konnectivity-agent-7ffsb\" (UID: \"b07dc994-256a-44d4-8405-715a10a1feb6\") " pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgdg\" (UniqueName: \"kubernetes.io/projected/0e41f154-bc28-4655-9b4a-1a8aaea58911-kube-api-access-gsgdg\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38526ca8-ee16-4dfe-84d6-965b4546ab8f-dbus\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.414497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413948 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-k8s-cni-cncf-io\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.413990 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhbd\" (UniqueName: \"kubernetes.io/projected/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-kube-api-access-9rhbd\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414042 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-multus-certs\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnjr5\" (UniqueName: \"kubernetes.io/projected/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-kube-api-access-hnjr5\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-host-slash\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-etc-selinux\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-cnibin\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-netns\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-hostroot\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-node-log\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.415094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-iptables-alerter-script\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414299 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-node-log\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-kubernetes\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysctl-conf\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-run\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a30d648-865f-476d-99a2-40f26f72ab32-ovn-node-metrics-cert\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-device-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-cni-multus\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-host\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414493 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-cnibin\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38526ca8-ee16-4dfe-84d6-965b4546ab8f-kubelet-config\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-conf-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-systemd\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.415569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-systemd-units\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-ovnkube-script-lib\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414721 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414732 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-run-systemd\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-systemd-units\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c91f3cb3-36af-4a18-8e17-0da940e6ef85-host\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-socket-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-sys\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-socket-dir-parent\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-daemon-config\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-etc-kubernetes\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-kubelet\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b07dc994-256a-44d4-8405-715a10a1feb6-agent-certs\") pod \"konnectivity-agent-7ffsb\" (UID: \"b07dc994-256a-44d4-8405-715a10a1feb6\") " pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.414973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-cni-binary-copy\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.416072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.415014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a30d648-865f-476d-99a2-40f26f72ab32-host-kubelet\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.416626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.415002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.416626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.415057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.416626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.415084 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-os-release\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.416626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.415108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-kubelet\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.416626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.415189 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-env-overrides\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.416626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.415251 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-ovnkube-config\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.416626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.415489 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a30d648-865f-476d-99a2-40f26f72ab32-ovnkube-script-lib\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.418328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.418309 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a30d648-865f-476d-99a2-40f26f72ab32-ovn-node-metrics-cert\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.424441 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.424423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfwrl\" (UniqueName: \"kubernetes.io/projected/9a30d648-865f-476d-99a2-40f26f72ab32-kube-api-access-nfwrl\") pod \"ovnkube-node-zd2ck\" (UID: \"9a30d648-865f-476d-99a2-40f26f72ab32\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.518146 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.517949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-sys-fs\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.518146 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518047 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-tuned\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.518146 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518049 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-sys-fs\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-tmp\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysconfig\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-system-cni-dir\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-system-cni-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-cni-binary-copy\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-cni-bin\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11b603d-097d-4bca-b187-43a143774e00-tmp-dir\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-system-cni-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmv4\" (UniqueName: \"kubernetes.io/projected/ffa68804-e8e5-473d-8bc2-b502e45313f1-kube-api-access-hsmv4\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518554 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-system-cni-dir\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-cni-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518432 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysconfig\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-lib-modules\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xz7\" (UniqueName: \"kubernetes.io/projected/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-kube-api-access-z7xz7\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.518681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c91f3cb3-36af-4a18-8e17-0da940e6ef85-serviceca\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-modprobe-d\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-var-lib-kubelet\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b11b603d-097d-4bca-b187-43a143774e00-hosts-file\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghd7k\" (UniqueName: \"kubernetes.io/projected/c91f3cb3-36af-4a18-8e17-0da940e6ef85-kube-api-access-ghd7k\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518969 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-cni-bin\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.518990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9975j\" (UniqueName: \"kubernetes.io/projected/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-kube-api-access-9975j\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysctl-d\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-systemd\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-os-release\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4lx\" (UniqueName: \"kubernetes.io/projected/b11b603d-097d-4bca-b187-43a143774e00-kube-api-access-9c4lx\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519170 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-registration-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.519484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519168 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-var-lib-kubelet\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b07dc994-256a-44d4-8405-715a10a1feb6-konnectivity-ca\") pod \"konnectivity-agent-7ffsb\" (UID: \"b07dc994-256a-44d4-8405-715a10a1feb6\") " pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-lib-modules\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519466 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-cni-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519477 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11b603d-097d-4bca-b187-43a143774e00-tmp-dir\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgdg\" (UniqueName: \"kubernetes.io/projected/0e41f154-bc28-4655-9b4a-1a8aaea58911-kube-api-access-gsgdg\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38526ca8-ee16-4dfe-84d6-965b4546ab8f-dbus\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519643 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-k8s-cni-cncf-io\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38526ca8-ee16-4dfe-84d6-965b4546ab8f-dbus\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhbd\" (UniqueName: \"kubernetes.io/projected/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-kube-api-access-9rhbd\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519752 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-k8s-cni-cncf-io\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.519966 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-systemd\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c91f3cb3-36af-4a18-8e17-0da940e6ef85-serviceca\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-os-release\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.520294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520152 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysctl-d\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520219 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-registration-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520243 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-cni-binary-copy\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.520332 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b11b603d-097d-4bca-b187-43a143774e00-hosts-file\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.520453 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs podName:ffa68804-e8e5-473d-8bc2-b502e45313f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:15.020421883 +0000 UTC m=+2.062646126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs") pod "network-metrics-daemon-qhkrt" (UID: "ffa68804-e8e5-473d-8bc2-b502e45313f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-modprobe-d\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-multus-certs\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnjr5\" (UniqueName: \"kubernetes.io/projected/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-kube-api-access-hnjr5\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520739 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b07dc994-256a-44d4-8405-715a10a1feb6-konnectivity-ca\") pod \"konnectivity-agent-7ffsb\" (UID: \"b07dc994-256a-44d4-8405-715a10a1feb6\") " pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-multus-certs\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.520759 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.520789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-host-slash\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.521138 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.520837 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret podName:38526ca8-ee16-4dfe-84d6-965b4546ab8f nodeName:}" failed. No retries permitted until 2026-07-16 21:39:15.020820107 +0000 UTC m=+2.063044326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret") pod "global-pull-secret-syncer-db75d" (UID: "38526ca8-ee16-4dfe-84d6-965b4546ab8f") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521712 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-tmp\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-tuned\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-etc-selinux\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521780 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-host-slash\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-cnibin\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521842 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-netns\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521888 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-cnibin\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-hostroot\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-iptables-alerter-script\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521922 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-etc-selinux\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521933 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-run-netns\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-kubernetes\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.522022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysctl-conf\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521981 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-hostroot\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-run\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522022 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-device-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.521985 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-kubernetes\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-cni-multus\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-host\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522062 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-run\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-cnibin\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522092 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-device-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522096 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-etc-sysctl-conf\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-cni-multus\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-cnibin\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522124 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-host\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.523153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38526ca8-ee16-4dfe-84d6-965b4546ab8f-kubelet-config\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522172 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-conf-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38526ca8-ee16-4dfe-84d6-965b4546ab8f-kubelet-config\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522220 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c91f3cb3-36af-4a18-8e17-0da940e6ef85-host\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-conf-dir\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-socket-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522268 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c91f3cb3-36af-4a18-8e17-0da940e6ef85-host\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-sys\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-socket-dir-parent\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522335 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-sys\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-daemon-config\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-etc-kubernetes\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b07dc994-256a-44d4-8405-715a10a1feb6-agent-certs\") pod \"konnectivity-agent-7ffsb\" (UID: \"b07dc994-256a-44d4-8405-715a10a1feb6\") " pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522384 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-socket-dir-parent\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-socket-dir\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.524042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-cni-binary-copy\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522467 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-iptables-alerter-script\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-os-release\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-kubelet\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522594 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-host-var-lib-kubelet\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522665 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e41f154-bc28-4655-9b4a-1a8aaea58911-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522436 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-etc-kubernetes\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522732 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-os-release\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.522968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-cni-binary-copy\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.523053 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e41f154-bc28-4655-9b4a-1a8aaea58911-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.523191 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-multus-daemon-config\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.524918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.524769 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b07dc994-256a-44d4-8405-715a10a1feb6-agent-certs\") pod \"konnectivity-agent-7ffsb\" (UID: \"b07dc994-256a-44d4-8405-715a10a1feb6\") " pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:14.526999 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.526962 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:39:14.527120 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.527003 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:39:14.527120 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.527018 2573 projected.go:196] Error preparing data for projected volume kube-api-access-ll8q8 for pod openshift-network-diagnostics/network-check-target-4qbsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:14.527120 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.527072 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8 podName:8df22d6f-dd67-413e-a3cb-7eea26dc2e57 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:15.027055298 +0000 UTC m=+2.069279534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8q8" (UniqueName: "kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8") pod "network-check-target-4qbsw" (UID: "8df22d6f-dd67-413e-a3cb-7eea26dc2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:14.529000 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.528913 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgdg\" (UniqueName: \"kubernetes.io/projected/0e41f154-bc28-4655-9b4a-1a8aaea58911-kube-api-access-gsgdg\") pod \"multus-additional-cni-plugins-zbvvv\" (UID: \"0e41f154-bc28-4655-9b4a-1a8aaea58911\") " pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.529225 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.529206 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhbd\" (UniqueName: \"kubernetes.io/projected/831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30-kube-api-access-9rhbd\") pod \"iptables-alerter-4b4zv\" (UID: \"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30\") " pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.529328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.529307 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xz7\" (UniqueName: \"kubernetes.io/projected/f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc-kube-api-access-z7xz7\") pod \"tuned-wvcf7\" (UID: \"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc\") " pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.529449 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.529434 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4lx\" (UniqueName: \"kubernetes.io/projected/b11b603d-097d-4bca-b187-43a143774e00-kube-api-access-9c4lx\") pod \"node-resolver-5jktm\" (UID: \"b11b603d-097d-4bca-b187-43a143774e00\") " pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.530730 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.530711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9975j\" (UniqueName: \"kubernetes.io/projected/df7ad3f6-95e5-4b6f-ad8d-1a5250937072-kube-api-access-9975j\") pod \"aws-ebs-csi-driver-node-hcfwt\" (UID: \"df7ad3f6-95e5-4b6f-ad8d-1a5250937072\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.531019 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.531000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghd7k\" (UniqueName: \"kubernetes.io/projected/c91f3cb3-36af-4a18-8e17-0da940e6ef85-kube-api-access-ghd7k\") pod \"node-ca-knpg7\" (UID: \"c91f3cb3-36af-4a18-8e17-0da940e6ef85\") " pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.534090 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.534037 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmv4\" (UniqueName: \"kubernetes.io/projected/ffa68804-e8e5-473d-8bc2-b502e45313f1-kube-api-access-hsmv4\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:14.537551 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.537525 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnjr5\" (UniqueName: \"kubernetes.io/projected/07c2a3d6-7fad-4f35-88f2-b2b25cfc6541-kube-api-access-hnjr5\") pod \"multus-z4zlv\" (UID: \"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541\") " pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.634689 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.634659 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:39:14.648729 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.648694 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:14.655774 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.655750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jktm" Jul 16 21:39:14.661873 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.661848 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" Jul 16 21:39:14.671188 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.671125 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda159e8edac5a31d9fd33bd8965a7119b.slice/crio-dc6a609aa5eb16b7a3859dc604f6a2d74ebbf7fbce544d1636c722eb6a70c5a3 WatchSource:0}: Error finding container dc6a609aa5eb16b7a3859dc604f6a2d74ebbf7fbce544d1636c722eb6a70c5a3: Status 404 returned error can't find the container with id dc6a609aa5eb16b7a3859dc604f6a2d74ebbf7fbce544d1636c722eb6a70c5a3 Jul 16 21:39:14.675108 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.675083 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a30d648_865f_476d_99a2_40f26f72ab32.slice/crio-3cc6cd911fe845360b315a5b6c6de1ac4bdaa6fae55cee99993938f11f8653ed WatchSource:0}: Error finding container 3cc6cd911fe845360b315a5b6c6de1ac4bdaa6fae55cee99993938f11f8653ed: Status 404 returned error can't find the container with id 3cc6cd911fe845360b315a5b6c6de1ac4bdaa6fae55cee99993938f11f8653ed Jul 16 21:39:14.677254 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.677231 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:39:14.677809 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.677773 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11b603d_097d_4bca_b187_43a143774e00.slice/crio-21900364bc8ecda143b119e8c5b5799e84c43c0579980f7a11330b0a83c918ec WatchSource:0}: Error finding container 21900364bc8ecda143b119e8c5b5799e84c43c0579980f7a11330b0a83c918ec: Status 404 returned error can't find the container with id 21900364bc8ecda143b119e8c5b5799e84c43c0579980f7a11330b0a83c918ec Jul 16 21:39:14.679468 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.679439 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad611d48b80ded02db93237655788291.slice/crio-f8b534dc95e1804c8fa0fb897a70c15340ed32818e9abc2761f953f237e161ad WatchSource:0}: Error finding container f8b534dc95e1804c8fa0fb897a70c15340ed32818e9abc2761f953f237e161ad: Status 404 returned error can't find the container with id f8b534dc95e1804c8fa0fb897a70c15340ed32818e9abc2761f953f237e161ad Jul 16 21:39:14.682665 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.682632 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8adb3b8_fc07_4fcc_ab3a_3e8eff5f3fdc.slice/crio-c3e31faa7d34aed69b2bb729caccba1b9ab401a87c38785a6db58bda8111feca WatchSource:0}: Error finding container c3e31faa7d34aed69b2bb729caccba1b9ab401a87c38785a6db58bda8111feca: Status 404 returned error can't find the container with id c3e31faa7d34aed69b2bb729caccba1b9ab401a87c38785a6db58bda8111feca Jul 16 21:39:14.687687 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.687666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4b4zv" Jul 16 21:39:14.697655 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.697623 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831e3efa_0ddb_4bfa_a7b4_2a5ac2447b30.slice/crio-85319a85993419ffc8775ff468e944cdf369a0565bdef501e59f08740b219fbe WatchSource:0}: Error finding container 85319a85993419ffc8775ff468e944cdf369a0565bdef501e59f08740b219fbe: Status 404 returned error can't find the container with id 85319a85993419ffc8775ff468e944cdf369a0565bdef501e59f08740b219fbe Jul 16 21:39:14.706060 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.706030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:14.710628 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.710574 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:39:14.715002 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.714978 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07dc994_256a_44d4_8405_715a10a1feb6.slice/crio-c62a3ae7f95d972d455743a6fc50363b8118d6acaca8a27b7bda120cb6578cca WatchSource:0}: Error finding container c62a3ae7f95d972d455743a6fc50363b8118d6acaca8a27b7bda120cb6578cca: Status 404 returned error can't find the container with id c62a3ae7f95d972d455743a6fc50363b8118d6acaca8a27b7bda120cb6578cca Jul 16 21:39:14.718306 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.718285 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" Jul 16 21:39:14.727123 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.727098 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e41f154_bc28_4655_9b4a_1a8aaea58911.slice/crio-c25de9e0c5a2d773e9455f81294ea6db07de3d8157a88899631dd21b9eb156df WatchSource:0}: Error finding container c25de9e0c5a2d773e9455f81294ea6db07de3d8157a88899631dd21b9eb156df: Status 404 returned error can't find the container with id c25de9e0c5a2d773e9455f81294ea6db07de3d8157a88899631dd21b9eb156df Jul 16 21:39:14.737148 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.737125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" Jul 16 21:39:14.743635 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.743609 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z4zlv" Jul 16 21:39:14.747035 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.747007 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7ad3f6_95e5_4b6f_ad8d_1a5250937072.slice/crio-bd2dd0a88c968c6cd1b5490385fab42bed872be6cc5808fda2eacdb0856e3a1e WatchSource:0}: Error finding container bd2dd0a88c968c6cd1b5490385fab42bed872be6cc5808fda2eacdb0856e3a1e: Status 404 returned error can't find the container with id bd2dd0a88c968c6cd1b5490385fab42bed872be6cc5808fda2eacdb0856e3a1e Jul 16 21:39:14.747035 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:14.747028 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knpg7" Jul 16 21:39:14.754546 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.754514 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c2a3d6_7fad_4f35_88f2_b2b25cfc6541.slice/crio-068385a0436c9f1ead7b7f6aa316b4eb3d205ba900a87f60e6a4b2eb95bc028f WatchSource:0}: Error finding container 068385a0436c9f1ead7b7f6aa316b4eb3d205ba900a87f60e6a4b2eb95bc028f: Status 404 returned error can't find the container with id 068385a0436c9f1ead7b7f6aa316b4eb3d205ba900a87f60e6a4b2eb95bc028f Jul 16 21:39:14.758893 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:39:14.758866 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91f3cb3_36af_4a18_8e17_0da940e6ef85.slice/crio-d10aaaf14386d22f55150069e135cc2f946a1fd6ed79390fcb075cc98ce8f027 WatchSource:0}: Error finding container d10aaaf14386d22f55150069e135cc2f946a1fd6ed79390fcb075cc98ce8f027: Status 404 returned error can't find the container with id d10aaaf14386d22f55150069e135cc2f946a1fd6ed79390fcb075cc98ce8f027 Jul 16 21:39:14.760601 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.760580 2573 kuberuntime_manager.go:1459] "Unhandled Error" err="container node-ca start failed in pod node-ca-knpg7_openshift-image-registry(c91f3cb3-36af-4a18-8e17-0da940e6ef85): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 16 21:39:14.760659 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:14.760615 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-image-registry/node-ca-knpg7" podUID="c91f3cb3-36af-4a18-8e17-0da940e6ef85" Jul 16 21:39:15.026281 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.026187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:15.026281 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.026249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:15.026477 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.026360 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:15.026477 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.026380 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:15.026477 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.026433 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs podName:ffa68804-e8e5-473d-8bc2-b502e45313f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:16.026416681 +0000 UTC m=+3.068640903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs") pod "network-metrics-daemon-qhkrt" (UID: "ffa68804-e8e5-473d-8bc2-b502e45313f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:15.026477 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.026449 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret podName:38526ca8-ee16-4dfe-84d6-965b4546ab8f nodeName:}" failed. No retries permitted until 2026-07-16 21:39:16.026442202 +0000 UTC m=+3.068666425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret") pod "global-pull-secret-syncer-db75d" (UID: "38526ca8-ee16-4dfe-84d6-965b4546ab8f") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:15.126857 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.126820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:15.127039 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.127005 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:39:15.127039 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.127029 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:39:15.127039 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.127039 2573 projected.go:196] Error preparing data for projected volume kube-api-access-ll8q8 for pod openshift-network-diagnostics/network-check-target-4qbsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:15.127178 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.127099 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8 podName:8df22d6f-dd67-413e-a3cb-7eea26dc2e57 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:16.127083092 +0000 UTC m=+3.169307308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8q8" (UniqueName: "kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8") pod "network-check-target-4qbsw" (UID: "8df22d6f-dd67-413e-a3cb-7eea26dc2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:15.364583 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.364407 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:34:15 +0000 UTC" deadline="2027-12-14 15:24:00.070361225 +0000 UTC" Jul 16 21:39:15.364583 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.364454 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12377h44m44.70591321s" Jul 16 21:39:15.537569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.537526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:15.537797 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.537689 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:15.552969 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.552897 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" event={"ID":"df7ad3f6-95e5-4b6f-ad8d-1a5250937072","Type":"ContainerStarted","Data":"bd2dd0a88c968c6cd1b5490385fab42bed872be6cc5808fda2eacdb0856e3a1e"} Jul 16 21:39:15.557215 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.557077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerStarted","Data":"c25de9e0c5a2d773e9455f81294ea6db07de3d8157a88899631dd21b9eb156df"} Jul 16 21:39:15.571225 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.571176 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ffsb" event={"ID":"b07dc994-256a-44d4-8405-715a10a1feb6","Type":"ContainerStarted","Data":"c62a3ae7f95d972d455743a6fc50363b8118d6acaca8a27b7bda120cb6578cca"} Jul 16 21:39:15.575185 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.575073 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4b4zv" event={"ID":"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30","Type":"ContainerStarted","Data":"85319a85993419ffc8775ff468e944cdf369a0565bdef501e59f08740b219fbe"} Jul 16 21:39:15.588832 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.588784 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"3cc6cd911fe845360b315a5b6c6de1ac4bdaa6fae55cee99993938f11f8653ed"} Jul 16 21:39:15.596252 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.596026 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" event={"ID":"a159e8edac5a31d9fd33bd8965a7119b","Type":"ContainerStarted","Data":"dc6a609aa5eb16b7a3859dc604f6a2d74ebbf7fbce544d1636c722eb6a70c5a3"} Jul 16 21:39:15.605632 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.605582 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knpg7" event={"ID":"c91f3cb3-36af-4a18-8e17-0da940e6ef85","Type":"ContainerStarted","Data":"d10aaaf14386d22f55150069e135cc2f946a1fd6ed79390fcb075cc98ce8f027"} Jul 16 21:39:15.614303 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:15.613860 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b8c58be4906c5077240bfb7d1315ca7709d0915b03b2b7020d68d6959102cf8\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-image-registry/node-ca-knpg7" podUID="c91f3cb3-36af-4a18-8e17-0da940e6ef85" Jul 16 21:39:15.614303 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.614243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" event={"ID":"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc","Type":"ContainerStarted","Data":"c3e31faa7d34aed69b2bb729caccba1b9ab401a87c38785a6db58bda8111feca"} Jul 16 21:39:15.624744 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.624575 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" event={"ID":"ad611d48b80ded02db93237655788291","Type":"ContainerStarted","Data":"f8b534dc95e1804c8fa0fb897a70c15340ed32818e9abc2761f953f237e161ad"} Jul 16 21:39:15.629763 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.629711 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jktm" event={"ID":"b11b603d-097d-4bca-b187-43a143774e00","Type":"ContainerStarted","Data":"21900364bc8ecda143b119e8c5b5799e84c43c0579980f7a11330b0a83c918ec"} Jul 16 21:39:15.639765 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.639724 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z4zlv" event={"ID":"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541","Type":"ContainerStarted","Data":"068385a0436c9f1ead7b7f6aa316b4eb3d205ba900a87f60e6a4b2eb95bc028f"} Jul 16 21:39:15.762921 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:15.762887 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:39:16.041566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:16.041467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:16.041566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:16.041544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:16.041831 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.041688 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:16.041831 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.041757 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret podName:38526ca8-ee16-4dfe-84d6-965b4546ab8f nodeName:}" failed. No retries permitted until 2026-07-16 21:39:18.04173803 +0000 UTC m=+5.083962250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret") pod "global-pull-secret-syncer-db75d" (UID: "38526ca8-ee16-4dfe-84d6-965b4546ab8f") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:16.042197 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.042175 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:16.042279 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.042240 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs podName:ffa68804-e8e5-473d-8bc2-b502e45313f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:18.042221505 +0000 UTC m=+5.084445723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs") pod "network-metrics-daemon-qhkrt" (UID: "ffa68804-e8e5-473d-8bc2-b502e45313f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:16.142867 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:16.142812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:16.143160 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.143044 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:39:16.143160 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.143069 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:39:16.143160 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.143084 2573 projected.go:196] Error preparing data for projected volume kube-api-access-ll8q8 for pod openshift-network-diagnostics/network-check-target-4qbsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:16.143160 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.143147 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8 podName:8df22d6f-dd67-413e-a3cb-7eea26dc2e57 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:18.14312703 +0000 UTC m=+5.185351251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8q8" (UniqueName: "kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8") pod "network-check-target-4qbsw" (UID: "8df22d6f-dd67-413e-a3cb-7eea26dc2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:16.331472 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:16.331183 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:39:16.365597 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:16.365479 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:34:15 +0000 UTC" deadline="2028-03-05 14:40:24.94154699 +0000 UTC" Jul 16 21:39:16.365597 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:16.365560 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14345h1m8.575991759s" Jul 16 21:39:16.534911 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:16.534872 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:16.535103 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.534994 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:16.535390 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:16.535366 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:16.535506 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:16.535474 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:17.541010 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:17.540153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:17.541010 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:17.540299 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:18.064573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:18.063736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:18.064573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:18.063813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:18.064573 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.064001 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:18.064573 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.064077 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret podName:38526ca8-ee16-4dfe-84d6-965b4546ab8f nodeName:}" failed. No retries permitted until 2026-07-16 21:39:22.064056745 +0000 UTC m=+9.106280967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret") pod "global-pull-secret-syncer-db75d" (UID: "38526ca8-ee16-4dfe-84d6-965b4546ab8f") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:18.064573 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.064484 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:18.064573 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.064534 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs podName:ffa68804-e8e5-473d-8bc2-b502e45313f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:22.064516286 +0000 UTC m=+9.106740507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs") pod "network-metrics-daemon-qhkrt" (UID: "ffa68804-e8e5-473d-8bc2-b502e45313f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:18.165272 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:18.165227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:18.165534 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.165428 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:39:18.165534 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.165450 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:39:18.165534 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.165464 2573 projected.go:196] Error preparing data for projected volume kube-api-access-ll8q8 for pod openshift-network-diagnostics/network-check-target-4qbsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:18.165534 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.165530 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8 podName:8df22d6f-dd67-413e-a3cb-7eea26dc2e57 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:22.165509745 +0000 UTC m=+9.207733965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8q8" (UniqueName: "kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8") pod "network-check-target-4qbsw" (UID: "8df22d6f-dd67-413e-a3cb-7eea26dc2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:18.535854 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:18.535122 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:18.535854 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.535290 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:18.535854 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:18.535389 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:18.535854 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:18.535512 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:19.536189 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:19.535613 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:19.536189 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:19.535756 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:20.534726 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:20.534682 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:20.534726 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:20.534724 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:20.535012 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:20.534869 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:20.535072 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:20.535003 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:21.534962 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:21.534917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:21.535440 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:21.535072 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:22.102723 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:22.102680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:22.102927 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:22.102759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:22.103001 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.102933 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:22.103059 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.102999 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret podName:38526ca8-ee16-4dfe-84d6-965b4546ab8f nodeName:}" failed. No retries permitted until 2026-07-16 21:39:30.102980212 +0000 UTC m=+17.145204437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret") pod "global-pull-secret-syncer-db75d" (UID: "38526ca8-ee16-4dfe-84d6-965b4546ab8f") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:22.103122 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.103093 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:22.103182 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.103145 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs podName:ffa68804-e8e5-473d-8bc2-b502e45313f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:30.103130156 +0000 UTC m=+17.145354388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs") pod "network-metrics-daemon-qhkrt" (UID: "ffa68804-e8e5-473d-8bc2-b502e45313f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:22.203868 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:22.203829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:22.204070 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.204019 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:39:22.204070 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.204053 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:39:22.204070 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.204066 2573 projected.go:196] Error preparing data for projected volume kube-api-access-ll8q8 for pod openshift-network-diagnostics/network-check-target-4qbsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:22.204239 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.204127 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8 podName:8df22d6f-dd67-413e-a3cb-7eea26dc2e57 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:30.204106588 +0000 UTC m=+17.246330808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8q8" (UniqueName: "kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8") pod "network-check-target-4qbsw" (UID: "8df22d6f-dd67-413e-a3cb-7eea26dc2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:22.535959 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:22.535156 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:22.535959 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.535373 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:22.535959 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:22.535753 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:22.536557 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:22.535951 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:23.536323 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:23.535847 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:23.536323 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:23.535964 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:24.534937 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:24.534870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:24.535170 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:24.534875 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:24.535170 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:24.535033 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:24.535170 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:24.535135 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:25.538368 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:25.538336 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:25.538887 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:25.538463 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:26.534819 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:26.534784 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:26.535002 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:26.534921 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:26.535002 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:26.534965 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:26.535119 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:26.535053 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:27.534456 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:27.534424 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:27.534898 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:27.534558 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:28.534700 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:28.534653 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:28.535138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:28.534707 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:28.535138 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:28.534805 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:28.535138 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:28.534924 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:29.535575 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:29.535533 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:29.536039 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:29.535669 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:30.166913 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:30.166875 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:30.166913 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:30.166931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:30.167151 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.167054 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:30.167151 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.167058 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:30.167151 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.167115 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret podName:38526ca8-ee16-4dfe-84d6-965b4546ab8f nodeName:}" failed. No retries permitted until 2026-07-16 21:39:46.167099783 +0000 UTC m=+33.209324002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret") pod "global-pull-secret-syncer-db75d" (UID: "38526ca8-ee16-4dfe-84d6-965b4546ab8f") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:30.167151 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.167130 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs podName:ffa68804-e8e5-473d-8bc2-b502e45313f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:46.167123286 +0000 UTC m=+33.209347502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs") pod "network-metrics-daemon-qhkrt" (UID: "ffa68804-e8e5-473d-8bc2-b502e45313f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:30.268068 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:30.268032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:30.268283 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.268260 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:39:30.268365 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.268293 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:39:30.268365 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.268307 2573 projected.go:196] Error preparing data for projected volume kube-api-access-ll8q8 for pod openshift-network-diagnostics/network-check-target-4qbsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:30.268437 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.268375 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8 podName:8df22d6f-dd67-413e-a3cb-7eea26dc2e57 nodeName:}" failed. No retries permitted until 2026-07-16 21:39:46.268354419 +0000 UTC m=+33.310578640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8q8" (UniqueName: "kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8") pod "network-check-target-4qbsw" (UID: "8df22d6f-dd67-413e-a3cb-7eea26dc2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:30.535485 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:30.535387 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:30.535673 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:30.535387 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:30.535673 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.535536 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:30.535673 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:30.535634 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:31.535338 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:31.535288 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:31.535553 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:31.535432 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:32.535303 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:32.535266 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:32.535303 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:32.535301 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:32.535853 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:32.535410 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:32.535853 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:32.535548 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:33.535760 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:33.535726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:33.536149 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:33.535824 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:34.534914 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:34.534877 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:34.535091 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:34.534877 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:34.535091 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:34.535010 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:34.535091 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:34.535053 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:35.539264 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.538901 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:35.539759 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:35.539345 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:35.698702 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.698670 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" event={"ID":"ad611d48b80ded02db93237655788291","Type":"ContainerStarted","Data":"34468c4fcfedfe54a1aa5ed42f047492d0c802ae8f6f88e181ce27bb709cd270"} Jul 16 21:39:35.700313 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.700285 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jktm" event={"ID":"b11b603d-097d-4bca-b187-43a143774e00","Type":"ContainerStarted","Data":"99b117810c37a6f0e6386745ce414fd6458103e9afcf930499b49247067ff1af"} Jul 16 21:39:35.701919 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.701895 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z4zlv" event={"ID":"07c2a3d6-7fad-4f35-88f2-b2b25cfc6541","Type":"ContainerStarted","Data":"5fbc42a60e6e0af60689c955990bbc222a13f98d39017e020986d115f05d1c71"} Jul 16 21:39:35.703338 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.703305 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" event={"ID":"df7ad3f6-95e5-4b6f-ad8d-1a5250937072","Type":"ContainerStarted","Data":"126f1a35a07f3ab94854df756b139087d2fbe2739ce8831240127223a7cd5832"} Jul 16 21:39:35.705335 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.705247 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e41f154-bc28-4655-9b4a-1a8aaea58911" containerID="4f1a6ffecce16bc72afb421dd4c47d6b207f967ba4b23d111a3a43d35407f29d" exitCode=0 Jul 16 21:39:35.705430 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.705337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerDied","Data":"4f1a6ffecce16bc72afb421dd4c47d6b207f967ba4b23d111a3a43d35407f29d"} Jul 16 21:39:35.707181 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.707145 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ffsb" event={"ID":"b07dc994-256a-44d4-8405-715a10a1feb6","Type":"ContainerStarted","Data":"dcfe85937f0048ec3f3fcffeafbdf4af8e980adb9f11b83557136329a913d26c"} Jul 16 21:39:35.710735 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.710707 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"61755cb26192b6b71d75554cb14b21fc0d94c349e426e9840e88214a00e1e9c9"} Jul 16 21:39:35.710814 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.710785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"e61c2b97ad4ba395025b639824f85b7b6c37fe5c9a62976aadeb3ebcd450a5d5"} Jul 16 21:39:35.710884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.710837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"a05a6199fbebb4bc11e7fddd94cf72511b2ff68c8097dfb0cd3a0b57a3be0a36"} Jul 16 21:39:35.710939 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.710890 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"672835eff653462499e4e613a79855e81c0a251918e9b2e8d2ec89de4a8c394b"} Jul 16 21:39:35.710989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.710936 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"382c5f3c337c782371a19c740b5e770b28f8864a5b78b27be3d7b7b674e1376f"} Jul 16 21:39:35.712356 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.712329 2573 generic.go:356] "Generic (PLEG): container finished" podID="a159e8edac5a31d9fd33bd8965a7119b" containerID="1c6b8ad5dd1f86f42cf88bf64d4e5e44f3f0c1f97737abafd661e55d08bceffa" exitCode=0 Jul 16 21:39:35.712444 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.712404 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" event={"ID":"a159e8edac5a31d9fd33bd8965a7119b","Type":"ContainerDied","Data":"1c6b8ad5dd1f86f42cf88bf64d4e5e44f3f0c1f97737abafd661e55d08bceffa"} Jul 16 21:39:35.714226 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.714150 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" event={"ID":"f8adb3b8-fc07-4fcc-ab3a-3e8eff5f3fdc","Type":"ContainerStarted","Data":"af0b4ba1bb521cd9de1dfc96c0011aea3f77ce1c38568de154bec668051611cb"} Jul 16 21:39:35.714401 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.714221 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" podStartSLOduration=20.714207069 podStartE2EDuration="20.714207069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:35.713119309 +0000 UTC m=+22.755343546" watchObservedRunningTime="2026-07-16 21:39:35.714207069 +0000 UTC m=+22.756431311" Jul 16 21:39:35.749689 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.749627 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5jktm" podStartSLOduration=1.464990418 podStartE2EDuration="21.749608858s" totalImagesPullingTime="20.28461844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:35.748838468 +0000 UTC m=+22.791062711" watchObservedRunningTime="2026-07-16 21:39:35.749608858 +0000 UTC m=+22.791833093" Jul 16 21:39:35.762972 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.762924 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7ffsb" podStartSLOduration=1.514859252 podStartE2EDuration="21.762909331s" totalImagesPullingTime="20.248050079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:35.762665306 +0000 UTC m=+22.804889544" watchObservedRunningTime="2026-07-16 21:39:35.762909331 +0000 UTC m=+22.805133568" Jul 16 21:39:35.780852 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.780795 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z4zlv" podStartSLOduration=1.505504231 podStartE2EDuration="21.780773746s" totalImagesPullingTime="20.275269515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:35.780574506 +0000 UTC m=+22.822798743" watchObservedRunningTime="2026-07-16 21:39:35.780773746 +0000 UTC m=+22.822997986" Jul 16 21:39:35.806890 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.806841 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wvcf7" podStartSLOduration=1.520770793 podStartE2EDuration="21.806824962s" totalImagesPullingTime="20.286054169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:35.806159727 +0000 UTC m=+22.848383965" watchObservedRunningTime="2026-07-16 21:39:35.806824962 +0000 UTC m=+22.849049200" Jul 16 21:39:35.925778 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:35.925685 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:36.003473 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.003383 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:36.210964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.210925 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:36.280906 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.280802 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:36.534849 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.534768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:36.534849 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.534793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:36.535066 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:36.534915 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:36.535066 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:36.535047 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:36.717964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.717933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4b4zv" event={"ID":"831e3efa-0ddb-4bfa-a7b4-2a5ac2447b30","Type":"ContainerStarted","Data":"130f029f9271657bf5f7d3c5f5f92d37d32690495906e99f979604fca7c98113"} Jul 16 21:39:36.720871 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.720842 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"eb3e0b22e5b71037b8b6e845404aec4babc0ad753afefd6bd3bb3b3be863d9e4"} Jul 16 21:39:36.733692 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.733644 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4b4zv" podStartSLOduration=6.6395542 podStartE2EDuration="22.733627142s" totalImagesPullingTime="16.094072942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:36.733155196 +0000 UTC m=+23.775379434" watchObservedRunningTime="2026-07-16 21:39:36.733627142 +0000 UTC m=+23.775851380" Jul 16 21:39:36.859583 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:36.859486 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:37.117023 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.116718 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 21:39:37.249607 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.249516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:37.350427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.350391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:37.407637 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.407460 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:39:37.116927347Z","UUID":"ab72b2eb-0ee1-473f-b66b-8b07cf077551","Handler":null,"Name":"","Endpoint":""} Jul 16 21:39:37.409178 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.409153 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 21:39:37.409283 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.409183 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 21:39:37.538005 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.537972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:37.538202 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:37.538083 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:37.725304 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.725209 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" event={"ID":"a159e8edac5a31d9fd33bd8965a7119b","Type":"ContainerStarted","Data":"aa7b9fa0a29238525742aefb525f1c23ac9b2afd40bf2cb66310853b6b87c814"} Jul 16 21:39:37.726762 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.726735 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knpg7" event={"ID":"c91f3cb3-36af-4a18-8e17-0da940e6ef85","Type":"ContainerStarted","Data":"f9b02cf403559f5d4a266e9080250dccdeee3ac2ef348dbe18f9cd70bca74369"} Jul 16 21:39:37.728542 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.728493 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" event={"ID":"df7ad3f6-95e5-4b6f-ad8d-1a5250937072","Type":"ContainerStarted","Data":"7484c727717acd009f0f5e457b76a720fc8a9ba751a6491a3b3619e38db1adb6"} Jul 16 21:39:37.743496 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.743438 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" podStartSLOduration=22.743423893 podStartE2EDuration="22.743423893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:37.742992119 +0000 UTC m=+24.785216357" watchObservedRunningTime="2026-07-16 21:39:37.743423893 +0000 UTC m=+24.785648130" Jul 16 21:39:37.761386 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.761331 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-knpg7" podStartSLOduration=1.596199226 podStartE2EDuration="23.761316555s" totalImagesPullingTime="22.165117329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-16 21:39:37.761307111 +0000 UTC m=+24.803531350" watchObservedRunningTime="2026-07-16 21:39:37.761316555 +0000 UTC m=+24.803540795" Jul 16 21:39:37.833024 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.832991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:37.934330 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:37.934271 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:38.535454 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:38.535421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:38.535614 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:38.535421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:38.535614 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:38.535579 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:38.535727 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:38.535647 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:38.735457 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:38.735416 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"d747832400d796e75ef89639f4e44747427e312d6cecabd2244396fa64a2cc70"} Jul 16 21:39:38.737814 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:38.737725 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" event={"ID":"df7ad3f6-95e5-4b6f-ad8d-1a5250937072","Type":"ContainerStarted","Data":"7d6dafdb6ff520d897cae9245286f1f32e08b15bbce883b5bb33bc1e1ac46c1b"} Jul 16 21:39:38.755535 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:38.755474 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcfwt" podStartSLOduration=1.604460797 podStartE2EDuration="24.755456948s" totalImagesPullingTime="23.150996151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:38.755200109 +0000 UTC m=+25.797424348" watchObservedRunningTime="2026-07-16 21:39:38.755456948 +0000 UTC m=+25.797681224" Jul 16 21:39:39.534788 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:39.534750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:39.534984 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:39.534900 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:39.614042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:39.614004 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:39.614652 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:39.614638 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:39.739898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:39.739831 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:39.740370 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:39.740351 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7ffsb" Jul 16 21:39:39.800652 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:39.800611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:40.535328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:40.535298 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:40.535545 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:40.535299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:40.535545 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:40.535411 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:40.535678 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:40.535544 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:41.138189 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.138154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:41.194342 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.194302 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:41.534905 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.534815 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:41.535055 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:41.534952 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:41.747507 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.747342 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" event={"ID":"9a30d648-865f-476d-99a2-40f26f72ab32","Type":"ContainerStarted","Data":"313485994e198f950a14b1f0b9221a43f125fe0b3339449e4fc841ab235435fb"} Jul 16 21:39:41.751231 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.751144 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:41.751231 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.751210 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:41.751366 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.751254 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:41.769245 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.769215 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:41.771001 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.770977 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:39:41.785920 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.785869 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" podStartSLOduration=7.464742015 podStartE2EDuration="27.78585029s" totalImagesPullingTime="20.321108275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:41.784991093 +0000 UTC m=+28.827215354" watchObservedRunningTime="2026-07-16 21:39:41.78585029 +0000 UTC m=+28.828074527" Jul 16 21:39:41.788613 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.788548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:41.868879 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.868841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:41.951206 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:41.951178 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:42.171700 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.171670 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:42.271748 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.271713 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:42.372752 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.372718 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:42.436282 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.436207 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:42.535116 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.535078 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:42.535116 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.535099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:42.535364 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:42.535180 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:42.535364 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:42.535290 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:42.617322 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.617282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:42.720512 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.720406 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:42.753097 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.753059 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e41f154-bc28-4655-9b4a-1a8aaea58911" containerID="2ea24b0b99e6b4dcfc8c6a2e918c60e26245fc6fbe320afb56be61e584e84d62" exitCode=0 Jul 16 21:39:42.754696 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:42.754665 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerDied","Data":"2ea24b0b99e6b4dcfc8c6a2e918c60e26245fc6fbe320afb56be61e584e84d62"} Jul 16 21:39:43.106949 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.106910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:43.180539 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.180492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:43.234963 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.234937 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jktm_b11b603d-097d-4bca-b187-43a143774e00/dns-node-resolver/0.log" Jul 16 21:39:43.251956 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.251923 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:43.323459 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.323424 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:43.535480 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.535451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:43.535659 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:43.535558 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:43.738475 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.738295 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhkrt"] Jul 16 21:39:43.738663 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.738601 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:43.738739 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:43.738717 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:43.738962 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.738944 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-db75d"] Jul 16 21:39:43.739024 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.739016 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:43.739101 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:43.739086 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:43.741388 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.741365 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4qbsw"] Jul 16 21:39:43.758001 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.757968 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerStarted","Data":"975343cf839eada75cab173792aad00f48e7c48743742fc91eaa459ed6234d85"} Jul 16 21:39:43.758001 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:43.758002 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:43.758194 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:43.758130 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:44.240012 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:44.239974 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:44.295135 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:44.295104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:44.435234 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:44.435159 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knpg7_c91f3cb3-36af-4a18-8e17-0da940e6ef85/node-ca/0.log" Jul 16 21:39:44.763334 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:44.763301 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e41f154-bc28-4655-9b4a-1a8aaea58911" containerID="975343cf839eada75cab173792aad00f48e7c48743742fc91eaa459ed6234d85" exitCode=0 Jul 16 21:39:44.763633 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:44.763412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerDied","Data":"975343cf839eada75cab173792aad00f48e7c48743742fc91eaa459ed6234d85"} Jul 16 21:39:45.243483 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:45.243443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:45.534665 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:45.534577 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:45.534824 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:45.534711 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:45.534824 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:45.534580 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:45.534916 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:45.534823 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:45.534916 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:45.534577 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:45.534916 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:45.534896 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:46.185336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:46.185309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:46.185424 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:46.185359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:46.185485 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:46.185471 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:46.185552 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:46.185482 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:46.185552 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:46.185545 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret podName:38526ca8-ee16-4dfe-84d6-965b4546ab8f nodeName:}" failed. No retries permitted until 2026-07-16 21:40:18.18551192 +0000 UTC m=+65.227736136 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret") pod "global-pull-secret-syncer-db75d" (UID: "38526ca8-ee16-4dfe-84d6-965b4546ab8f") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:39:46.185639 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:46.185560 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs podName:ffa68804-e8e5-473d-8bc2-b502e45313f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:40:18.18555423 +0000 UTC m=+65.227778446 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs") pod "network-metrics-daemon-qhkrt" (UID: "ffa68804-e8e5-473d-8bc2-b502e45313f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:39:46.286009 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:46.285966 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:46.286454 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:46.286100 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:39:46.286454 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:46.286115 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:39:46.286454 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:46.286125 2573 projected.go:196] Error preparing data for projected volume kube-api-access-ll8q8 for pod openshift-network-diagnostics/network-check-target-4qbsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:46.286454 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:46.286177 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8 podName:8df22d6f-dd67-413e-a3cb-7eea26dc2e57 nodeName:}" failed. No retries permitted until 2026-07-16 21:40:18.286162577 +0000 UTC m=+65.328386793 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8q8" (UniqueName: "kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8") pod "network-check-target-4qbsw" (UID: "8df22d6f-dd67-413e-a3cb-7eea26dc2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:39:46.522353 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:46.522307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:46.770489 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:46.770397 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e41f154-bc28-4655-9b4a-1a8aaea58911" containerID="991f8c0e0620d21c6554285c04c34161a544ce880d8e0e413f5dcca7c146652e" exitCode=0 Jul 16 21:39:46.770489 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:46.770450 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerDied","Data":"991f8c0e0620d21c6554285c04c34161a544ce880d8e0e413f5dcca7c146652e"} Jul 16 21:39:47.535046 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:47.535009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:47.535514 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:47.535013 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:47.535514 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:47.535147 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:47.535514 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:47.535207 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:47.535514 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:47.535289 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:47.535514 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:47.535361 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:48.537345 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:48.537308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:48.641392 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:48.641359 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:48.755600 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:48.755559 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:48.855135 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:48.855093 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:48.955297 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:48.955257 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:49.453923 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:49.453895 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:49.535043 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:49.535007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:49.535249 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:49.535007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:49.535249 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:49.535142 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:49.535249 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:49.535018 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:49.535249 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:49.535204 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:49.535461 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:49.535313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:49.771218 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:49.771133 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:49.871076 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:49.871036 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:50.296144 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:50.296106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:50.448235 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:50.448195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:50.631550 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:50.631451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:51.036016 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:51.035927 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:51.255092 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:51.255062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:51.535410 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:51.535374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:51.535410 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:51.535407 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:51.535680 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:51.535373 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:51.535680 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:51.535551 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:51.535680 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:51.535625 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:51.535843 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:51.535683 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:51.633376 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:51.633322 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:52.033868 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:52.033831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:52.233444 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:52.233240 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:52.503250 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:52.503214 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:52.559066 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:52.559037 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:52.789934 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:52.789840 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e41f154-bc28-4655-9b4a-1a8aaea58911" containerID="71190353677ece665aab893e7b689bb1c0866cca6d48db178ea2f85f696f547e" exitCode=0 Jul 16 21:39:52.789934 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:52.789900 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerDied","Data":"71190353677ece665aab893e7b689bb1c0866cca6d48db178ea2f85f696f547e"} Jul 16 21:39:52.842864 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:52.842825 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:53.031879 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:53.031848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:53.092595 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:53.092569 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:53.537003 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:53.536279 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:53.537003 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:53.536436 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:53.537003 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:53.536709 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:53.537003 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:53.536831 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:53.538310 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:53.537744 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:53.538310 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:53.537866 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:53.641111 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:53.641073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:53.794836 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:53.794808 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e41f154-bc28-4655-9b4a-1a8aaea58911" containerID="e7bb5014941d1fe5df615fbf3cf0590a5d34ca195e5df5ee3fa0a4ae55808e72" exitCode=0 Jul 16 21:39:53.794972 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:53.794865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerDied","Data":"e7bb5014941d1fe5df615fbf3cf0590a5d34ca195e5df5ee3fa0a4ae55808e72"} Jul 16 21:39:54.030725 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:54.030689 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:54.250651 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:54.250558 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:54.634686 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:54.634647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:54.800614 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:54.800579 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" event={"ID":"0e41f154-bc28-4655-9b4a-1a8aaea58911","Type":"ContainerStarted","Data":"ee05a81077ae20c87ba65fd6e26ae7b82b4337301320090a266c4874e0cc9da8"} Jul 16 21:39:54.822815 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:54.822768 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zbvvv" podStartSLOduration=7.031608731 podStartE2EDuration="40.822750763s" totalImagesPullingTime="33.791142032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:54.821784913 +0000 UTC m=+41.864009150" watchObservedRunningTime="2026-07-16 21:39:54.822750763 +0000 UTC m=+41.864974978" Jul 16 21:39:54.841540 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:54.841490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:55.241892 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:55.241856 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:55.535541 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:55.535438 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:55.535541 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:55.535459 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:55.535703 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:55.535610 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:55.535703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:55.535656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:55.535774 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:55.535725 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:55.535809 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:55.535788 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:55.835017 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:55.834987 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:56.036651 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:56.036615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:56.237775 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:56.237697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:56.335236 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:56.335206 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:56.439767 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:56.439737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:57.534921 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:57.534888 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:57.535318 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:57.534888 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:57.535318 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:57.535004 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:39:57.535318 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:57.535114 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:57.535318 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:57.534891 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:57.535318 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:57.535220 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:57.651175 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:57.651135 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:57.844971 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:57.844931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:58.041055 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:58.041018 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:58.238112 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:58.238010 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:58.436096 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:58.436064 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:59.037346 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:59.037313 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:59.244336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:59.244305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:39:59.537785 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:59.537758 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:39:59.537965 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:59.537851 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:39:59.537965 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:59.537928 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:39:59.538054 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:39:59.537959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:39:59.538054 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:59.538032 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:39:59.538184 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:39:59.538165 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:40:01.535277 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:01.535239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:01.535746 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:01.535340 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:40:01.535746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:01.535239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:40:01.535746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:01.535239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:40:01.535746 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:01.535406 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:40:01.535746 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:01.535505 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:40:03.535904 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:03.535866 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:40:03.536286 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:03.535964 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:40:03.536286 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:03.535969 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:40:03.536286 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:03.536061 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:40:03.536286 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:03.536111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:03.536286 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:03.536159 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:40:03.882677 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:03.882639 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:03.960005 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:03.959973 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:04.155265 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:04.155167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:04.215238 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:04.215205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:04.365691 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:04.365662 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:04.584899 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:04.584861 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:04.738416 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:04.738388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:04.839234 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:04.839152 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:05.349172 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:05.349139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:05.534669 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:05.534626 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:05.534669 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:05.534656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:40:05.534893 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:05.534626 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:40:05.534893 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:05.534747 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:40:05.534893 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:05.534800 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:40:05.534893 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:05.534878 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:40:05.553394 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:05.553359 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:05.772338 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:05.772250 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:05.919162 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:05.919127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:07.534535 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:07.534488 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:40:07.535084 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:07.534488 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:07.535084 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:07.534637 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhkrt" podUID="ffa68804-e8e5-473d-8bc2-b502e45313f1" Jul 16 21:40:07.535084 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:07.534490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:40:07.535084 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:07.534685 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4qbsw" podUID="8df22d6f-dd67-413e-a3cb-7eea26dc2e57" Jul 16 21:40:07.535084 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:07.534761 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db75d" podUID="38526ca8-ee16-4dfe-84d6-965b4546ab8f" Jul 16 21:40:08.088740 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.088702 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:08.263030 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.262989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:08.363555 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.363432 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:08.825831 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.825812 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeReady" Jul 16 21:40:08.826192 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.825933 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 21:40:08.879260 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.879229 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gdc8v"] Jul 16 21:40:08.881108 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.881092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:08.883630 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.883604 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 21:40:08.883786 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.883699 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 21:40:08.883786 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.883712 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vpjhj\"" Jul 16 21:40:08.883786 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.883777 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 21:40:08.883966 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.883823 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 21:40:08.886345 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.886323 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mwxc9"] Jul 16 21:40:08.888238 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.888221 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:08.890719 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.890698 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2qh2q\"" Jul 16 21:40:08.890829 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.890699 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 21:40:08.890959 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.890947 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 21:40:08.898305 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.898279 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gdc8v"] Jul 16 21:40:08.900839 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.900818 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mwxc9"] Jul 16 21:40:08.985837 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.985803 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qgf2x"] Jul 16 21:40:08.987687 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.987669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgf2x" Jul 16 21:40:08.993460 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.993435 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 21:40:08.993634 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.993552 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wdq25\"" Jul 16 21:40:08.994369 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.994327 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 21:40:08.994465 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:08.994450 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 21:40:09.004174 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.004147 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgf2x"] Jul 16 21:40:09.022012 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.021975 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:09.041231 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbsc\" (UniqueName: \"kubernetes.io/projected/b1872486-e142-4f26-8afb-aed610a1db52-kube-api-access-9sbsc\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.041231 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041234 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e1f0037-10ff-44e8-9b40-7979eace45a5-metrics-tls\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.041443 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041257 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e1f0037-10ff-44e8-9b40-7979eace45a5-tmp-dir\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.041443 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1872486-e142-4f26-8afb-aed610a1db52-crio-socket\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.041443 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1872486-e142-4f26-8afb-aed610a1db52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.041443 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e1f0037-10ff-44e8-9b40-7979eace45a5-config-volume\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.041443 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zhd\" (UniqueName: \"kubernetes.io/projected/1e1f0037-10ff-44e8-9b40-7979eace45a5-kube-api-access-f6zhd\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.041643 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1872486-e142-4f26-8afb-aed610a1db52-data-volume\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.041643 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.041470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1872486-e142-4f26-8afb-aed610a1db52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.142608 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.142512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbsc\" (UniqueName: \"kubernetes.io/projected/b1872486-e142-4f26-8afb-aed610a1db52-kube-api-access-9sbsc\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.142608 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.142564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e1f0037-10ff-44e8-9b40-7979eace45a5-metrics-tls\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.142783 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.142680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e1f0037-10ff-44e8-9b40-7979eace45a5-tmp-dir\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.142783 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.142726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1872486-e142-4f26-8afb-aed610a1db52-crio-socket\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.142783 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.142757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1872486-e142-4f26-8afb-aed610a1db52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.142904 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.142784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d86a9fa6-dda8-43f2-9330-fa89e82d0e61-cert\") pod \"ingress-canary-qgf2x\" (UID: \"d86a9fa6-dda8-43f2-9330-fa89e82d0e61\") " pod="openshift-ingress-canary/ingress-canary-qgf2x" Jul 16 21:40:09.142979 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.142928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1872486-e142-4f26-8afb-aed610a1db52-crio-socket\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.143020 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.142986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e1f0037-10ff-44e8-9b40-7979eace45a5-config-volume\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.143020 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.143013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zhd\" (UniqueName: \"kubernetes.io/projected/1e1f0037-10ff-44e8-9b40-7979eace45a5-kube-api-access-f6zhd\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.143125 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.143030 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e1f0037-10ff-44e8-9b40-7979eace45a5-tmp-dir\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.143125 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.143034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1872486-e142-4f26-8afb-aed610a1db52-data-volume\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.143125 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.143096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92bb\" (UniqueName: \"kubernetes.io/projected/d86a9fa6-dda8-43f2-9330-fa89e82d0e61-kube-api-access-m92bb\") pod \"ingress-canary-qgf2x\" (UID: \"d86a9fa6-dda8-43f2-9330-fa89e82d0e61\") " pod="openshift-ingress-canary/ingress-canary-qgf2x" Jul 16 21:40:09.143268 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.143131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1872486-e142-4f26-8afb-aed610a1db52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.143365 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.143346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1872486-e142-4f26-8afb-aed610a1db52-data-volume\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.143616 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.143596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e1f0037-10ff-44e8-9b40-7979eace45a5-config-volume\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.144269 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.144239 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1872486-e142-4f26-8afb-aed610a1db52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.146699 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.146682 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e1f0037-10ff-44e8-9b40-7979eace45a5-metrics-tls\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.146938 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.146711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1872486-e142-4f26-8afb-aed610a1db52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.150338 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.150316 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbsc\" (UniqueName: \"kubernetes.io/projected/b1872486-e142-4f26-8afb-aed610a1db52-kube-api-access-9sbsc\") pod \"insights-runtime-extractor-gdc8v\" (UID: \"b1872486-e142-4f26-8afb-aed610a1db52\") " pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.150586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.150568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zhd\" (UniqueName: \"kubernetes.io/projected/1e1f0037-10ff-44e8-9b40-7979eace45a5-kube-api-access-f6zhd\") pod \"dns-default-mwxc9\" (UID: \"1e1f0037-10ff-44e8-9b40-7979eace45a5\") " pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.190616 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.190580 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gdc8v" Jul 16 21:40:09.197337 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.197309 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:09.244418 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.244379 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d86a9fa6-dda8-43f2-9330-fa89e82d0e61-cert\") pod \"ingress-canary-qgf2x\" (UID: \"d86a9fa6-dda8-43f2-9330-fa89e82d0e61\") " pod="openshift-ingress-canary/ingress-canary-qgf2x" Jul 16 21:40:09.244639 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.244485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92bb\" (UniqueName: \"kubernetes.io/projected/d86a9fa6-dda8-43f2-9330-fa89e82d0e61-kube-api-access-m92bb\") pod \"ingress-canary-qgf2x\" (UID: \"d86a9fa6-dda8-43f2-9330-fa89e82d0e61\") " pod="openshift-ingress-canary/ingress-canary-qgf2x" Jul 16 21:40:09.246954 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.246925 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d86a9fa6-dda8-43f2-9330-fa89e82d0e61-cert\") pod \"ingress-canary-qgf2x\" (UID: \"d86a9fa6-dda8-43f2-9330-fa89e82d0e61\") " pod="openshift-ingress-canary/ingress-canary-qgf2x" Jul 16 21:40:09.252256 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.252208 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92bb\" (UniqueName: \"kubernetes.io/projected/d86a9fa6-dda8-43f2-9330-fa89e82d0e61-kube-api-access-m92bb\") pod \"ingress-canary-qgf2x\" (UID: \"d86a9fa6-dda8-43f2-9330-fa89e82d0e61\") " pod="openshift-ingress-canary/ingress-canary-qgf2x" Jul 16 21:40:09.297167 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.296797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgf2x" Jul 16 21:40:09.341248 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.341213 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gdc8v"] Jul 16 21:40:09.345973 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:09.345946 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1872486_e142_4f26_8afb_aed610a1db52.slice/crio-54b9b7386f68e3eebaf2b0743bece5fc5107690c56bb90fef2ec0eb8e168d456 WatchSource:0}: Error finding container 54b9b7386f68e3eebaf2b0743bece5fc5107690c56bb90fef2ec0eb8e168d456: Status 404 returned error can't find the container with id 54b9b7386f68e3eebaf2b0743bece5fc5107690c56bb90fef2ec0eb8e168d456 Jul 16 21:40:09.353971 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.353939 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mwxc9"] Jul 16 21:40:09.368123 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.368089 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:09.429931 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.429850 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgf2x"] Jul 16 21:40:09.432798 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:09.432765 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86a9fa6_dda8_43f2_9330_fa89e82d0e61.slice/crio-eea6a47066c742a385b93dc16a1c9d6761047c029aa7d89472e52a6ca34913ba WatchSource:0}: Error finding container eea6a47066c742a385b93dc16a1c9d6761047c029aa7d89472e52a6ca34913ba: Status 404 returned error can't find the container with id eea6a47066c742a385b93dc16a1c9d6761047c029aa7d89472e52a6ca34913ba Jul 16 21:40:09.534737 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.534697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:40:09.534919 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.534826 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:40:09.534919 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.534697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:09.537157 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.537129 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:40:09.537278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.537255 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:40:09.537408 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.537392 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:40:09.537584 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.537563 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:40:09.537670 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.537613 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bjvwm\"" Jul 16 21:40:09.537809 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.537789 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cd5nk\"" Jul 16 21:40:09.835234 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.835189 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgf2x" event={"ID":"d86a9fa6-dda8-43f2-9330-fa89e82d0e61","Type":"ContainerStarted","Data":"eea6a47066c742a385b93dc16a1c9d6761047c029aa7d89472e52a6ca34913ba"} Jul 16 21:40:09.836546 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.836503 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mwxc9" event={"ID":"1e1f0037-10ff-44e8-9b40-7979eace45a5","Type":"ContainerStarted","Data":"2c8e69cf5fe7b1423f22c65985b120bf8f5e92504dbb94ef91d1348e08bf295c"} Jul 16 21:40:09.837746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:09.837714 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdc8v" event={"ID":"b1872486-e142-4f26-8afb-aed610a1db52","Type":"ContainerStarted","Data":"54b9b7386f68e3eebaf2b0743bece5fc5107690c56bb90fef2ec0eb8e168d456"} Jul 16 21:40:11.845917 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:11.845870 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mwxc9" event={"ID":"1e1f0037-10ff-44e8-9b40-7979eace45a5","Type":"ContainerStarted","Data":"088ea5c938b3667fd4498a5e6d58a6deb50e06c0822ddac28cc6780d9397fcf4"} Jul 16 21:40:11.846361 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:11.845942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mwxc9" event={"ID":"1e1f0037-10ff-44e8-9b40-7979eace45a5","Type":"ContainerStarted","Data":"f9277aca202c281f222b6aac22dbea2a731ffaa5ff358d596d32b6ae3af8b488"} Jul 16 21:40:11.846361 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:11.846018 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:11.865241 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:11.865183 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mwxc9" podStartSLOduration=1.961445677 podStartE2EDuration="3.865165458s" totalImagesPullingTime="1.903719781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:11.864122602 +0000 UTC m=+58.906346842" watchObservedRunningTime="2026-07-16 21:40:11.865165458 +0000 UTC m=+58.907389673" Jul 16 21:40:12.729573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:12.729512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:12.825343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:12.825307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:12.894651 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:12.894607 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6"] Jul 16 21:40:12.895095 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:12.894701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:12.902458 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:12.902427 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" Jul 16 21:40:12.904957 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:12.904930 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dz2ns\"" Jul 16 21:40:12.905086 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:12.904974 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 21:40:12.907392 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:12.907365 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6"] Jul 16 21:40:13.079041 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.079007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-xxqr6\" (UID: \"2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" Jul 16 21:40:13.180283 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.180243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-xxqr6\" (UID: \"2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" Jul 16 21:40:13.180479 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:13.180403 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 16 21:40:13.180568 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:13.180489 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a-tls-certificates podName:2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a nodeName:}" failed. No retries permitted until 2026-07-16 21:40:13.680458547 +0000 UTC m=+60.722682771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-xxqr6" (UID: "2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a") : secret "prometheus-operator-admission-webhook-tls" not found Jul 16 21:40:13.300046 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.300013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:13.397710 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.397621 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55b8669d59-gvc6x"] Jul 16 21:40:13.397920 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.397895 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:13.413365 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.413328 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b8669d59-gvc6x"] Jul 16 21:40:13.413535 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.413461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.416927 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.416209 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 21:40:13.416927 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.416756 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rq4nh\"" Jul 16 21:40:13.417325 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.417306 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 21:40:13.417728 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.417705 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 21:40:13.417941 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.417925 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 21:40:13.420487 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.418577 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 21:40:13.420487 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.418668 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 21:40:13.420487 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.418865 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 21:40:13.468137 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.468099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:13.541094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.541055 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:13.583059 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.583015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-config\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.583059 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.583059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-serving-cert\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.583281 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.583087 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-oauth-serving-cert\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.583281 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.583187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-oauth-config\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.583281 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.583229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7z95\" (UniqueName: \"kubernetes.io/projected/b33ec7ed-281a-4dff-be63-3f6e7453a984-kube-api-access-x7z95\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.583281 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.583245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-service-ca\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.684446 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.684352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-xxqr6\" (UID: \"2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" Jul 16 21:40:13.684446 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.684399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-config\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.684446 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.684431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-serving-cert\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.684907 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.684457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-oauth-serving-cert\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.684907 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.684495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-oauth-config\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.684907 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.684722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7z95\" (UniqueName: \"kubernetes.io/projected/b33ec7ed-281a-4dff-be63-3f6e7453a984-kube-api-access-x7z95\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.684907 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.684768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-service-ca\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.685316 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.685258 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-oauth-serving-cert\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.685406 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.685367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-config\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.685830 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.685810 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-service-ca\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.687389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.687361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-oauth-config\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.687558 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.687516 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-xxqr6\" (UID: \"2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" Jul 16 21:40:13.687727 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.687702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-serving-cert\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.693751 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.693724 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7z95\" (UniqueName: \"kubernetes.io/projected/b33ec7ed-281a-4dff-be63-3f6e7453a984-kube-api-access-x7z95\") pod \"console-55b8669d59-gvc6x\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.725477 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.725440 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:13.774454 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.774423 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd2ck" Jul 16 21:40:13.816867 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.816825 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dz2ns\"" Jul 16 21:40:13.824950 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:13.824916 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" Jul 16 21:40:14.644127 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:14.644098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:14.699451 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:14.699406 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:15.038807 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:15.038713 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:15.096957 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:15.096925 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:15.858895 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:15.858860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdc8v" event={"ID":"b1872486-e142-4f26-8afb-aed610a1db52","Type":"ContainerStarted","Data":"216358e813d517c5c8d6ab9cc72584d9d71a014da33383250b33b7dd538e841a"} Jul 16 21:40:15.859358 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:15.859339 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b8669d59-gvc6x"] Jul 16 21:40:15.860569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:15.860542 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgf2x" event={"ID":"d86a9fa6-dda8-43f2-9330-fa89e82d0e61","Type":"ContainerStarted","Data":"85dd5e067cbff9e76dd561400b28b974ca9821fb236389158059104c20c40bc8"} Jul 16 21:40:15.867324 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:15.865140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6"] Jul 16 21:40:15.872001 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:15.871948 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eabe3d7_5ebe_43b4_98b7_8ca39a1c6c2a.slice/crio-53baa58b78d0987092df779a1060f60df583139dcaad5195ebf83593f9e90c83 WatchSource:0}: Error finding container 53baa58b78d0987092df779a1060f60df583139dcaad5195ebf83593f9e90c83: Status 404 returned error can't find the container with id 53baa58b78d0987092df779a1060f60df583139dcaad5195ebf83593f9e90c83 Jul 16 21:40:15.877768 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:15.877719 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qgf2x" podStartSLOduration=1.634288972 podStartE2EDuration="7.877702814s" totalImagesPullingTime="6.243413842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:15.876989826 +0000 UTC m=+62.919214068" watchObservedRunningTime="2026-07-16 21:40:15.877702814 +0000 UTC m=+62.919927031" Jul 16 21:40:16.866088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:16.866045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" event={"ID":"2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a","Type":"ContainerStarted","Data":"53baa58b78d0987092df779a1060f60df583139dcaad5195ebf83593f9e90c83"} Jul 16 21:40:16.868113 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:16.868033 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b8669d59-gvc6x" event={"ID":"b33ec7ed-281a-4dff-be63-3f6e7453a984","Type":"ContainerStarted","Data":"874bf5c3d4bba2942bbc01ae1a609b397cdf51022ac13616422578a2489a109e"} Jul 16 21:40:17.872604 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:17.872562 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" event={"ID":"2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a","Type":"ContainerStarted","Data":"f2943e0f05aaee8d0fe4c60ff5e478dc38c5c5fb4b1f68ac956b5b31d817b46a"} Jul 16 21:40:17.873081 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:17.873062 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" Jul 16 21:40:17.878873 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:17.878846 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" Jul 16 21:40:17.888333 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:17.888258 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xxqr6" podStartSLOduration=4.511656032 podStartE2EDuration="5.888237298s" totalImagesPullingTime="1.376581266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:17.887457993 +0000 UTC m=+64.929682227" watchObservedRunningTime="2026-07-16 21:40:17.888237298 +0000 UTC m=+64.930461537" Jul 16 21:40:18.223124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.223036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:40:18.223285 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.223137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:40:18.225827 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.225798 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:40:18.225827 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.225798 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:40:18.236790 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.236721 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa68804-e8e5-473d-8bc2-b502e45313f1-metrics-certs\") pod \"network-metrics-daemon-qhkrt\" (UID: \"ffa68804-e8e5-473d-8bc2-b502e45313f1\") " pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:40:18.236790 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.236743 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38526ca8-ee16-4dfe-84d6-965b4546ab8f-original-pull-secret\") pod \"global-pull-secret-syncer-db75d\" (UID: \"38526ca8-ee16-4dfe-84d6-965b4546ab8f\") " pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:40:18.248639 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.248605 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bjvwm\"" Jul 16 21:40:18.250657 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.250627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db75d" Jul 16 21:40:18.256667 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.256644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhkrt" Jul 16 21:40:18.323455 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.323414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:18.326369 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.326341 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:40:18.339726 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.339694 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:40:18.347712 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.347683 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8q8\" (UniqueName: \"kubernetes.io/projected/8df22d6f-dd67-413e-a3cb-7eea26dc2e57-kube-api-access-ll8q8\") pod \"network-check-target-4qbsw\" (UID: \"8df22d6f-dd67-413e-a3cb-7eea26dc2e57\") " pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:18.558298 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.558264 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cd5nk\"" Jul 16 21:40:18.566490 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.566456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:18.922321 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.922228 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:18.968945 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.968914 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg"] Jul 16 21:40:18.971403 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.971377 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:18.974024 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.973992 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 21:40:18.974426 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.974404 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 21:40:18.974575 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.974456 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 21:40:18.975043 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.975021 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 21:40:18.975426 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.975405 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zpmb7\"" Jul 16 21:40:18.975550 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.975464 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 21:40:18.985051 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.985023 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg"] Jul 16 21:40:18.999686 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:18.999645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:19.027944 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.027911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c17c3042-2778-4bd3-a6e4-48bd3910f356-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.028126 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.027975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.028126 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.028004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.028126 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.028032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzqw\" (UniqueName: \"kubernetes.io/projected/c17c3042-2778-4bd3-a6e4-48bd3910f356-kube-api-access-lzzqw\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.128596 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.128557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzqw\" (UniqueName: \"kubernetes.io/projected/c17c3042-2778-4bd3-a6e4-48bd3910f356-kube-api-access-lzzqw\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.128801 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.128638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c17c3042-2778-4bd3-a6e4-48bd3910f356-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.128801 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.128696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.128801 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.128726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.128963 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:19.128838 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 16 21:40:19.128963 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:19.128928 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-tls podName:c17c3042-2778-4bd3-a6e4-48bd3910f356 nodeName:}" failed. No retries permitted until 2026-07-16 21:40:19.628892319 +0000 UTC m=+66.671116548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-rr5fg" (UID: "c17c3042-2778-4bd3-a6e4-48bd3910f356") : secret "prometheus-operator-tls" not found Jul 16 21:40:19.129448 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.129420 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c17c3042-2778-4bd3-a6e4-48bd3910f356-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.131773 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.131749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.138094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.138062 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzqw\" (UniqueName: \"kubernetes.io/projected/c17c3042-2778-4bd3-a6e4-48bd3910f356-kube-api-access-lzzqw\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.363566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.363508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:19.632215 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.632122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.634959 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.634931 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c17c3042-2778-4bd3-a6e4-48bd3910f356-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rr5fg\" (UID: \"c17c3042-2778-4bd3-a6e4-48bd3910f356\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:19.740551 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.740492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:19.841029 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.840976 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:19.882419 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:19.882327 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" Jul 16 21:40:20.592058 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.592023 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhkrt"] Jul 16 21:40:20.597378 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:20.597326 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa68804_e8e5_473d_8bc2_b502e45313f1.slice/crio-ecd635607f57c8deb93cb25238281ed64d180ac182207f90ab7d20f722d60f54 WatchSource:0}: Error finding container ecd635607f57c8deb93cb25238281ed64d180ac182207f90ab7d20f722d60f54: Status 404 returned error can't find the container with id ecd635607f57c8deb93cb25238281ed64d180ac182207f90ab7d20f722d60f54 Jul 16 21:40:20.604234 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.604140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4qbsw"] Jul 16 21:40:20.609324 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:20.609291 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df22d6f_dd67_413e_a3cb_7eea26dc2e57.slice/crio-96b16895511ef4671cebe92fa75144dc83b6d8a10134426eb917a187781315a9 WatchSource:0}: Error finding container 96b16895511ef4671cebe92fa75144dc83b6d8a10134426eb917a187781315a9: Status 404 returned error can't find the container with id 96b16895511ef4671cebe92fa75144dc83b6d8a10134426eb917a187781315a9 Jul 16 21:40:20.833981 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.833813 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-db75d"] Jul 16 21:40:20.836911 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:20.836867 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38526ca8_ee16_4dfe_84d6_965b4546ab8f.slice/crio-f152d3252ec68862ec662ef94dfdb8ae2b97e0e45eb76a9e827cd996d020469c WatchSource:0}: Error finding container f152d3252ec68862ec662ef94dfdb8ae2b97e0e45eb76a9e827cd996d020469c: Status 404 returned error can't find the container with id f152d3252ec68862ec662ef94dfdb8ae2b97e0e45eb76a9e827cd996d020469c Jul 16 21:40:20.837058 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.836953 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg"] Jul 16 21:40:20.839829 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:20.839801 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17c3042_2778_4bd3_a6e4_48bd3910f356.slice/crio-8578dee8db36cc623e2669de0673baed0ac1ccaefd1c70388c2b90125c515012 WatchSource:0}: Error finding container 8578dee8db36cc623e2669de0673baed0ac1ccaefd1c70388c2b90125c515012: Status 404 returned error can't find the container with id 8578dee8db36cc623e2669de0673baed0ac1ccaefd1c70388c2b90125c515012 Jul 16 21:40:20.883101 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.883060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4qbsw" event={"ID":"8df22d6f-dd67-413e-a3cb-7eea26dc2e57","Type":"ContainerStarted","Data":"96b16895511ef4671cebe92fa75144dc83b6d8a10134426eb917a187781315a9"} Jul 16 21:40:20.884142 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.884115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" event={"ID":"c17c3042-2778-4bd3-a6e4-48bd3910f356","Type":"ContainerStarted","Data":"8578dee8db36cc623e2669de0673baed0ac1ccaefd1c70388c2b90125c515012"} Jul 16 21:40:20.885149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.885126 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhkrt" event={"ID":"ffa68804-e8e5-473d-8bc2-b502e45313f1","Type":"ContainerStarted","Data":"ecd635607f57c8deb93cb25238281ed64d180ac182207f90ab7d20f722d60f54"} Jul 16 21:40:20.886116 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.886097 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-db75d" event={"ID":"38526ca8-ee16-4dfe-84d6-965b4546ab8f","Type":"ContainerStarted","Data":"f152d3252ec68862ec662ef94dfdb8ae2b97e0e45eb76a9e827cd996d020469c"} Jul 16 21:40:20.887648 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.887631 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdc8v" event={"ID":"b1872486-e142-4f26-8afb-aed610a1db52","Type":"ContainerStarted","Data":"4fa883620154cf079eea59867ac678242139532a91217a79d971f1208628efae"} Jul 16 21:40:20.889094 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.889075 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b8669d59-gvc6x" event={"ID":"b33ec7ed-281a-4dff-be63-3f6e7453a984","Type":"ContainerStarted","Data":"1ee6192f4579d815136be37f9d535dd36bb368326464a62cadc65dcfad3ba714"} Jul 16 21:40:20.907840 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:20.907788 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55b8669d59-gvc6x" podStartSLOduration=3.357610021 podStartE2EDuration="7.907771324s" totalImagesPullingTime="4.550161303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:20.90682059 +0000 UTC m=+67.949044828" watchObservedRunningTime="2026-07-16 21:40:20.907771324 +0000 UTC m=+67.949995561" Jul 16 21:40:21.851867 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:21.851806 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mwxc9" Jul 16 21:40:21.973981 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:21.972932 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55f648488-qr5ht"] Jul 16 21:40:21.976833 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:21.976488 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:21.986187 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:21.986147 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 21:40:21.991428 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:21.991394 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f648488-qr5ht"] Jul 16 21:40:22.045278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.045018 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-trusted-ca-bundle\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.045278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.045070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-serving-cert\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.045278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.045099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-oauth-config\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.045278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.045130 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2vq\" (UniqueName: \"kubernetes.io/projected/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-kube-api-access-6d2vq\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.045278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.045176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-service-ca\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.045278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.045211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-config\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.045747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.045303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-oauth-serving-cert\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.147364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.146192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-service-ca\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.147364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.146250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-config\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.147364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.146291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-oauth-serving-cert\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.147364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.146328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-trusted-ca-bundle\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.147364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.146360 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-serving-cert\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.147364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.146384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-oauth-config\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.147364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.146414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2vq\" (UniqueName: \"kubernetes.io/projected/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-kube-api-access-6d2vq\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.148649 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.147847 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-trusted-ca-bundle\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.148649 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.148173 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-oauth-serving-cert\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.148649 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.148312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-config\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.148807 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.148660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-service-ca\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.151313 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.151270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-serving-cert\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.151969 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.151946 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-oauth-config\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.164280 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.164244 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2vq\" (UniqueName: \"kubernetes.io/projected/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-kube-api-access-6d2vq\") pod \"console-55f648488-qr5ht\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.294538 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.294389 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:22.511747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.511211 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f648488-qr5ht"] Jul 16 21:40:22.522978 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:22.522943 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab77e64_ef70_47c0_abef_0cf2954cdd3d.slice/crio-b9090ea728151994ab569b160c8ab6a1f0b2b4223747a18833b7f0cace1f772e WatchSource:0}: Error finding container b9090ea728151994ab569b160c8ab6a1f0b2b4223747a18833b7f0cace1f772e: Status 404 returned error can't find the container with id b9090ea728151994ab569b160c8ab6a1f0b2b4223747a18833b7f0cace1f772e Jul 16 21:40:22.765908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.765827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:22.901340 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.900874 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f648488-qr5ht" event={"ID":"0ab77e64-ef70-47c0-abef-0cf2954cdd3d","Type":"ContainerStarted","Data":"eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84"} Jul 16 21:40:22.901340 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.900946 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f648488-qr5ht" event={"ID":"0ab77e64-ef70-47c0-abef-0cf2954cdd3d","Type":"ContainerStarted","Data":"b9090ea728151994ab569b160c8ab6a1f0b2b4223747a18833b7f0cace1f772e"} Jul 16 21:40:22.903456 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.903419 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" event={"ID":"c17c3042-2778-4bd3-a6e4-48bd3910f356","Type":"ContainerStarted","Data":"e667e633d821a09ea58b8cd4d402898bcfae5c4e6b763063b90c970de571db19"} Jul 16 21:40:22.903635 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.903486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" event={"ID":"c17c3042-2778-4bd3-a6e4-48bd3910f356","Type":"ContainerStarted","Data":"4a563597815d58d387655d5a409993d29faec83ebe1ebefef5c50af44da34d05"} Jul 16 21:40:22.908277 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.907666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhkrt" event={"ID":"ffa68804-e8e5-473d-8bc2-b502e45313f1","Type":"ContainerStarted","Data":"4f2ffd995b7cc33db7b79ae5f9de1e78d6ada71fa357fd0e9ef390fef3db5dd7"} Jul 16 21:40:22.908277 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.907726 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhkrt" event={"ID":"ffa68804-e8e5-473d-8bc2-b502e45313f1","Type":"ContainerStarted","Data":"32f239fc4678d5dc4e835e79ad1d52cc9c62900c1c35e2afeb973249e18edc33"} Jul 16 21:40:22.919686 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.919620 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55f648488-qr5ht" podStartSLOduration=1.919596223 podStartE2EDuration="1.919596223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:22.918252767 +0000 UTC m=+69.960477018" watchObservedRunningTime="2026-07-16 21:40:22.919596223 +0000 UTC m=+69.961820462" Jul 16 21:40:22.954507 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.954270 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qhkrt" podStartSLOduration=67.754989447 podStartE2EDuration="1m8.954252313s" totalImagesPullingTime="1.199262866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:22.953883507 +0000 UTC m=+69.996107746" watchObservedRunningTime="2026-07-16 21:40:22.954252313 +0000 UTC m=+69.996476551" Jul 16 21:40:22.956131 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:22.955658 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rr5fg" podStartSLOduration=3.4316576469999998 podStartE2EDuration="4.955646269s" totalImagesPullingTime="1.523988622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:22.935119022 +0000 UTC m=+69.977343260" watchObservedRunningTime="2026-07-16 21:40:22.955646269 +0000 UTC m=+69.997870509" Jul 16 21:40:23.052379 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:23.052290 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:23.128007 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:23.127970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:23.589835 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:23.589783 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:23.655489 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:23.655458 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:23.726310 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:23.726222 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:23.726310 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:23.726284 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:23.728262 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:23.727919 2573 patch_prober.go:28] interesting pod/console-55b8669d59-gvc6x container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 16 21:40:23.728262 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:23.727976 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-55b8669d59-gvc6x" podUID="b33ec7ed-281a-4dff-be63-3f6e7453a984" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 16 21:40:25.081977 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.081937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.275831 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.275783 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.302451 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.302417 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj"] Jul 16 21:40:25.305919 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.305893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.309574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.309335 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 21:40:25.309574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.309341 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k7zd9\"" Jul 16 21:40:25.309820 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.309659 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 21:40:25.319751 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.319714 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj"] Jul 16 21:40:25.322470 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.322437 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d9jbv"] Jul 16 21:40:25.326065 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.326038 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.329117 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.328593 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 21:40:25.329117 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.328967 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 21:40:25.329117 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.329071 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-hqjsh"] Jul 16 21:40:25.329893 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.329238 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pm9ql\"" Jul 16 21:40:25.329893 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.329458 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 21:40:25.332824 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.332765 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.335786 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.335583 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 21:40:25.335786 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.335613 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 21:40:25.335916 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.335889 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kmp24\"" Jul 16 21:40:25.335964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.335942 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 21:40:25.347471 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.347441 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-hqjsh"] Jul 16 21:40:25.375882 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.375842 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.376149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46hx\" (UniqueName: \"kubernetes.io/projected/f9b83b31-e9fa-40eb-854d-05d789cb77cd-kube-api-access-v46hx\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.376226 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-root\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.376226 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rqj\" (UniqueName: \"kubernetes.io/projected/d6e14b29-ad65-44ed-8119-8333506ad460-kube-api-access-n7rqj\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.376314 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-accelerators-collector-config\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.376314 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376280 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b83b31-e9fa-40eb-854d-05d789cb77cd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.376414 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-sys\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.376414 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362547e8-d75f-4861-b842-d5aec9f80ba9-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.376414 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-textfile\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.376573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.376573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.376573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/362547e8-d75f-4861-b842-d5aec9f80ba9-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.376573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.376746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.376746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376625 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9fm9\" (UniqueName: \"kubernetes.io/projected/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-api-access-w9fm9\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.376746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.376869 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-tls\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.376869 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.376869 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-wtmp\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.376869 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.376847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6e14b29-ad65-44ed-8119-8333506ad460-metrics-client-ca\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.440830 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.440782 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.477691 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.477884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-wtmp\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.477884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477729 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6e14b29-ad65-44ed-8119-8333506ad460-metrics-client-ca\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.477884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46hx\" (UniqueName: \"kubernetes.io/projected/f9b83b31-e9fa-40eb-854d-05d789cb77cd-kube-api-access-v46hx\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.477884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-root\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.477884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rqj\" (UniqueName: \"kubernetes.io/projected/d6e14b29-ad65-44ed-8119-8333506ad460-kube-api-access-n7rqj\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.477884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-accelerators-collector-config\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.477884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b83b31-e9fa-40eb-854d-05d789cb77cd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-sys\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362547e8-d75f-4861-b842-d5aec9f80ba9-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477912 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-wtmp\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-textfile\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.477998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/362547e8-d75f-4861-b842-d5aec9f80ba9-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fm9\" (UniqueName: \"kubernetes.io/projected/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-api-access-w9fm9\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478150 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-root\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-tls\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.478276 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:25.478258 2573 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 16 21:40:25.479002 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:25.478316 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-tls podName:f9b83b31-e9fa-40eb-854d-05d789cb77cd nodeName:}" failed. No retries permitted until 2026-07-16 21:40:25.978296516 +0000 UTC m=+73.020520750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-q8jcj" (UID: "f9b83b31-e9fa-40eb-854d-05d789cb77cd") : secret "openshift-state-metrics-tls" not found Jul 16 21:40:25.479002 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:25.478367 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 21:40:25.479002 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:25.478425 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-tls podName:d6e14b29-ad65-44ed-8119-8333506ad460 nodeName:}" failed. No retries permitted until 2026-07-16 21:40:25.978410498 +0000 UTC m=+73.020634713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-tls") pod "node-exporter-d9jbv" (UID: "d6e14b29-ad65-44ed-8119-8333506ad460") : secret "node-exporter-tls" not found Jul 16 21:40:25.479002 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478446 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6e14b29-ad65-44ed-8119-8333506ad460-metrics-client-ca\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.479002 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478446 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-accelerators-collector-config\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.479002 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.478827 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/362547e8-d75f-4861-b842-d5aec9f80ba9-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.479432 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.479397 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362547e8-d75f-4861-b842-d5aec9f80ba9-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.479560 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.479480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6e14b29-ad65-44ed-8119-8333506ad460-sys\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.479560 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.479480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.479751 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.479731 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-textfile\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.480085 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.480054 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b83b31-e9fa-40eb-854d-05d789cb77cd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.481471 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.481444 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.481609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.481539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.481609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.481552 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.481609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.481571 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.487774 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.487715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46hx\" (UniqueName: \"kubernetes.io/projected/f9b83b31-e9fa-40eb-854d-05d789cb77cd-kube-api-access-v46hx\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.488617 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.488581 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rqj\" (UniqueName: \"kubernetes.io/projected/d6e14b29-ad65-44ed-8119-8333506ad460-kube-api-access-n7rqj\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.489566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.489541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fm9\" (UniqueName: \"kubernetes.io/projected/362547e8-d75f-4861-b842-d5aec9f80ba9-kube-api-access-w9fm9\") pod \"kube-state-metrics-56c5594674-hqjsh\" (UID: \"362547e8-d75f-4861-b842-d5aec9f80ba9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.539561 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.539503 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.639867 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.639740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.647625 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.647586 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" Jul 16 21:40:25.780085 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.780050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.881050 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.880988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.981694 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.981593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:25.981694 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.981667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-tls\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.981907 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.981711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:25.984737 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.984702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6e14b29-ad65-44ed-8119-8333506ad460-node-exporter-tls\") pod \"node-exporter-d9jbv\" (UID: \"d6e14b29-ad65-44ed-8119-8333506ad460\") " pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:25.984925 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:25.984747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9b83b31-e9fa-40eb-854d-05d789cb77cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-q8jcj\" (UID: \"f9b83b31-e9fa-40eb-854d-05d789cb77cd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:26.217204 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.217168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" Jul 16 21:40:26.240213 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.240131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d9jbv" Jul 16 21:40:26.342294 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.342260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:26.424226 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.424164 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:26.425437 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.425406 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:40:26.446981 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.446950 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:40:26.447169 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.447120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.449876 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.449688 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 21:40:26.450151 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.450126 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 21:40:26.450418 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.450399 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 21:40:26.450594 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.450445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 21:40:26.450979 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.450955 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 21:40:26.453309 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.451043 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 21:40:26.453309 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.451824 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 21:40:26.453309 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.452017 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 21:40:26.453309 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.452314 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 21:40:26.453309 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.452907 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7mwbr\"" Jul 16 21:40:26.486571 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52z4s\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-kube-api-access-52z4s\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.486743 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.486743 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-out\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.486743 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486727 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.486895 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.486895 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.486977 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486895 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.486977 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.487074 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.486981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.487074 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.487027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.487074 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.487045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.487074 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.487060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-web-config\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.487240 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.487095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.540927 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.540839 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:26.587986 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.587939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.587986 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.587985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52z4s\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-kube-api-access-52z4s\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588238 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588238 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-out\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588238 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588238 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588238 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588496 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588496 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:40:26.588386 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle podName:cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b nodeName:}" failed. No retries permitted until 2026-07-16 21:40:27.088360515 +0000 UTC m=+74.130584748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b") : configmap references non-existent config key: ca-bundle.crt Jul 16 21:40:26.588496 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588496 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588496 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588806 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588806 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.588806 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.588619 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-web-config\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.589483 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.589104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.591371 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.591290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-out\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.591663 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.591639 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-web-config\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.591771 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.591711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.591868 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.591840 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.591921 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.591903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.591961 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.591940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.592098 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.592079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.592240 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.592217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.593128 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.593109 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.596201 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.596176 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52z4s\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-kube-api-access-52z4s\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:26.642576 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:26.642511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:27.093766 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.093713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:27.094758 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.094732 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:27.363755 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.363662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:40:27.826710 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.826552 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj"] Jul 16 21:40:27.830690 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:27.830632 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b83b31_e9fa_40eb_854d_05d789cb77cd.slice/crio-5aadcb6e2aafe8dab6bc3e4f73c0eae1e8a1a012a89d0f7d778ebc1685c3ea59 WatchSource:0}: Error finding container 5aadcb6e2aafe8dab6bc3e4f73c0eae1e8a1a012a89d0f7d778ebc1685c3ea59: Status 404 returned error can't find the container with id 5aadcb6e2aafe8dab6bc3e4f73c0eae1e8a1a012a89d0f7d778ebc1685c3ea59 Jul 16 21:40:27.858431 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.858398 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:40:27.864573 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:27.864468 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2e7e76_cb3c_4141_b2d8_e6b33f8eef6b.slice/crio-33a1195252c7fa98919b5fd7e43c84f2edffe33fa15a7d67360b747451b1f9db WatchSource:0}: Error finding container 33a1195252c7fa98919b5fd7e43c84f2edffe33fa15a7d67360b747451b1f9db: Status 404 returned error can't find the container with id 33a1195252c7fa98919b5fd7e43c84f2edffe33fa15a7d67360b747451b1f9db Jul 16 21:40:27.872090 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.871999 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-hqjsh"] Jul 16 21:40:27.884878 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:27.884851 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362547e8_d75f_4861_b842_d5aec9f80ba9.slice/crio-4c0015eacf150fa51122011802133096e650016a85e2823aeb91ea60895a4e94 WatchSource:0}: Error finding container 4c0015eacf150fa51122011802133096e650016a85e2823aeb91ea60895a4e94: Status 404 returned error can't find the container with id 4c0015eacf150fa51122011802133096e650016a85e2823aeb91ea60895a4e94 Jul 16 21:40:27.930589 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.930498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4qbsw" event={"ID":"8df22d6f-dd67-413e-a3cb-7eea26dc2e57","Type":"ContainerStarted","Data":"12554dc68ef6cd2c2edb3c20556be9135d386b7343f7da71698d06d3a5467591"} Jul 16 21:40:27.930746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.930715 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:27.932282 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.932206 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerStarted","Data":"33a1195252c7fa98919b5fd7e43c84f2edffe33fa15a7d67360b747451b1f9db"} Jul 16 21:40:27.933473 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.933442 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" event={"ID":"362547e8-d75f-4861-b842-d5aec9f80ba9","Type":"ContainerStarted","Data":"4c0015eacf150fa51122011802133096e650016a85e2823aeb91ea60895a4e94"} Jul 16 21:40:27.935798 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.935771 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-db75d" event={"ID":"38526ca8-ee16-4dfe-84d6-965b4546ab8f","Type":"ContainerStarted","Data":"2e06bada1a7b036f7aad096adb1582c6b2762ebccb40a2e9dc68c9a576cf832b"} Jul 16 21:40:27.937394 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.937370 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d9jbv" event={"ID":"d6e14b29-ad65-44ed-8119-8333506ad460","Type":"ContainerStarted","Data":"28b9f617594475971ffdad9a6cc459c34626dd0094d448d53b6ad7f9c620830c"} Jul 16 21:40:27.940337 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.940310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdc8v" event={"ID":"b1872486-e142-4f26-8afb-aed610a1db52","Type":"ContainerStarted","Data":"dfc7b6a7d9f4cd07b01c699eabc335e497028cfc510ff7006cae54ba9195111b"} Jul 16 21:40:27.943508 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.943475 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" event={"ID":"f9b83b31-e9fa-40eb-854d-05d789cb77cd","Type":"ContainerStarted","Data":"af104216118baaf8d2d65b53b78389352d354c7ba3bcecba52765b0d04e86c9b"} Jul 16 21:40:27.943754 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.943728 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" event={"ID":"f9b83b31-e9fa-40eb-854d-05d789cb77cd","Type":"ContainerStarted","Data":"5aadcb6e2aafe8dab6bc3e4f73c0eae1e8a1a012a89d0f7d778ebc1685c3ea59"} Jul 16 21:40:27.966452 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.966401 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4qbsw" podStartSLOduration=66.924017292 podStartE2EDuration="1m13.966383294s" totalImagesPullingTime="7.042366002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:27.946792275 +0000 UTC m=+74.989016513" watchObservedRunningTime="2026-07-16 21:40:27.966383294 +0000 UTC m=+75.008607509" Jul 16 21:40:27.967168 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.967134 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gdc8v" podStartSLOduration=1.848429011 podStartE2EDuration="19.967119403s" totalImagesPullingTime="18.118690392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:40:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:27.966227417 +0000 UTC m=+75.008451714" watchObservedRunningTime="2026-07-16 21:40:27.967119403 +0000 UTC m=+75.009343642" Jul 16 21:40:27.980753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:27.980684 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-db75d" podStartSLOduration=67.16552633 podStartE2EDuration="1m13.980663583s" totalImagesPullingTime="6.815137253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:27.980605039 +0000 UTC m=+75.022829281" watchObservedRunningTime="2026-07-16 21:40:27.980663583 +0000 UTC m=+75.022887821" Jul 16 21:40:28.334211 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.334175 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:28.420874 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.420834 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:28.421412 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.421388 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv"] Jul 16 21:40:28.426615 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.426587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.429099 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.429070 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 16 21:40:28.429397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.429129 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 16 21:40:28.429397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.429070 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8dwzk\"" Jul 16 21:40:28.429397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.429346 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 16 21:40:28.429397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.429394 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-62682pkhigfk8\"" Jul 16 21:40:28.429664 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.429346 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 16 21:40:28.429664 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.429603 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 16 21:40:28.438472 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.438433 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv"] Jul 16 21:40:28.506000 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.505962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-tls\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.506156 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.506012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.506156 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.506069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.506156 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.506110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1357ede3-e5c3-403f-806f-258928451826-metrics-client-ca\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.506156 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.506132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqzj\" (UniqueName: \"kubernetes.io/projected/1357ede3-e5c3-403f-806f-258928451826-kube-api-access-fqqzj\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.506323 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.506192 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.506323 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.506222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-grpc-tls\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.506323 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.506243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.607438 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.607402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-grpc-tls\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.607623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.607446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.607623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.607483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-tls\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.607623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.607509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.607623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.607545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.607623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.607566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1357ede3-e5c3-403f-806f-258928451826-metrics-client-ca\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.607623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.607584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqzj\" (UniqueName: \"kubernetes.io/projected/1357ede3-e5c3-403f-806f-258928451826-kube-api-access-fqqzj\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.607926 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.607658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.608469 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.608434 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1357ede3-e5c3-403f-806f-258928451826-metrics-client-ca\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.610670 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.610623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-tls\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.610670 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.610649 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.610670 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.610635 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-grpc-tls\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.611071 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.611051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.611106 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.611051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.611106 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.611078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1357ede3-e5c3-403f-806f-258928451826-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.615268 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.615243 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqzj\" (UniqueName: \"kubernetes.io/projected/1357ede3-e5c3-403f-806f-258928451826-kube-api-access-fqqzj\") pod \"thanos-querier-6bd44f97fc-mvdsv\" (UID: \"1357ede3-e5c3-403f-806f-258928451826\") " pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.634542 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.634422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:28.740611 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.740564 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:28.916606 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.916462 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv"] Jul 16 21:40:28.922735 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:28.922699 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1357ede3_e5c3_403f_806f_258928451826.slice/crio-0df9b7e1893f514975c22552b9e0d6a703b75267010968c67dfcb98547b5d02e WatchSource:0}: Error finding container 0df9b7e1893f514975c22552b9e0d6a703b75267010968c67dfcb98547b5d02e: Status 404 returned error can't find the container with id 0df9b7e1893f514975c22552b9e0d6a703b75267010968c67dfcb98547b5d02e Jul 16 21:40:28.952479 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.952260 2573 generic.go:356] "Generic (PLEG): container finished" podID="d6e14b29-ad65-44ed-8119-8333506ad460" containerID="e1321bfd42094f1d8b42715befac1c234013274f3cc1e421afc7cab007ac8729" exitCode=0 Jul 16 21:40:28.952479 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.952407 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d9jbv" event={"ID":"d6e14b29-ad65-44ed-8119-8333506ad460","Type":"ContainerDied","Data":"e1321bfd42094f1d8b42715befac1c234013274f3cc1e421afc7cab007ac8729"} Jul 16 21:40:28.955444 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.955409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" event={"ID":"f9b83b31-e9fa-40eb-854d-05d789cb77cd","Type":"ContainerStarted","Data":"570e85536fc6919d8b458a9a20598b8f4a906f01cbc23d45b54be4493434848b"} Jul 16 21:40:28.957538 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:28.957477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" event={"ID":"1357ede3-e5c3-403f-806f-258928451826","Type":"ContainerStarted","Data":"0df9b7e1893f514975c22552b9e0d6a703b75267010968c67dfcb98547b5d02e"} Jul 16 21:40:29.628758 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.628721 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bbd464946-dpq6k"] Jul 16 21:40:29.634698 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.634659 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.638586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.638556 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 21:40:29.638586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.638572 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 21:40:29.638816 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.638580 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 21:40:29.646651 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.646621 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ksnjl\"" Jul 16 21:40:29.646819 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.646666 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 21:40:29.647088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.646939 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bkau5926jlacl\"" Jul 16 21:40:29.647382 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.647360 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bbd464946-dpq6k"] Jul 16 21:40:29.719003 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.718965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/69485f8e-111b-4abe-8669-549bb5bb6fe5-audit-log\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.719197 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.719014 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-client-ca-bundle\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.719197 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.719043 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/69485f8e-111b-4abe-8669-549bb5bb6fe5-metrics-server-audit-profiles\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.719197 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.719078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69485f8e-111b-4abe-8669-549bb5bb6fe5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.719197 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.719154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-secret-metrics-server-tls\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.719339 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.719244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-secret-metrics-server-client-certs\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.719339 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.719268 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnldv\" (UniqueName: \"kubernetes.io/projected/69485f8e-111b-4abe-8669-549bb5bb6fe5-kube-api-access-hnldv\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.820470 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.820428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-secret-metrics-server-tls\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.820733 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.820502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-secret-metrics-server-client-certs\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.820733 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.820552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnldv\" (UniqueName: \"kubernetes.io/projected/69485f8e-111b-4abe-8669-549bb5bb6fe5-kube-api-access-hnldv\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.820733 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.820618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/69485f8e-111b-4abe-8669-549bb5bb6fe5-audit-log\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.820733 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.820644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-client-ca-bundle\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.820733 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.820690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/69485f8e-111b-4abe-8669-549bb5bb6fe5-metrics-server-audit-profiles\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.820981 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.820736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69485f8e-111b-4abe-8669-549bb5bb6fe5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.821031 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.821010 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/69485f8e-111b-4abe-8669-549bb5bb6fe5-audit-log\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.821468 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.821445 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69485f8e-111b-4abe-8669-549bb5bb6fe5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.821786 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.821763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/69485f8e-111b-4abe-8669-549bb5bb6fe5-metrics-server-audit-profiles\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.823118 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.823097 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-secret-metrics-server-tls\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.823118 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.823110 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-client-ca-bundle\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.823260 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.823154 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/69485f8e-111b-4abe-8669-549bb5bb6fe5-secret-metrics-server-client-certs\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.829412 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.829375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnldv\" (UniqueName: \"kubernetes.io/projected/69485f8e-111b-4abe-8669-549bb5bb6fe5-kube-api-access-hnldv\") pod \"metrics-server-bbd464946-dpq6k\" (UID: \"69485f8e-111b-4abe-8669-549bb5bb6fe5\") " pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.947853 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.947607 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:29.968446 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.968405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerStarted","Data":"3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b"} Jul 16 21:40:29.970547 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.970494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" event={"ID":"362547e8-d75f-4861-b842-d5aec9f80ba9","Type":"ContainerStarted","Data":"0627bfc0e6514119b1636303fd01d178b807c65e1998d1ade929858107d9b629"} Jul 16 21:40:29.972282 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.972254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:29.973065 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.973043 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d9jbv" event={"ID":"d6e14b29-ad65-44ed-8119-8333506ad460","Type":"ContainerStarted","Data":"730ec94c08e6a0dac4c71361d7ef521671edb8afc856ab9329c56a26898bac6a"} Jul 16 21:40:29.975370 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:29.975347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" event={"ID":"f9b83b31-e9fa-40eb-854d-05d789cb77cd","Type":"ContainerStarted","Data":"2b37b9ff2b53b89dc0e43aa640affca8f48587279a3f5e76526224e5779fabe5"} Jul 16 21:40:30.020976 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.020614 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-q8jcj" podStartSLOduration=3.163061883 podStartE2EDuration="5.020589425s" totalImagesPullingTime="1.857527542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:30.018857528 +0000 UTC m=+77.061081774" watchObservedRunningTime="2026-07-16 21:40:30.020589425 +0000 UTC m=+77.062813667" Jul 16 21:40:30.126083 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.125844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:30.139351 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.139324 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8"] Jul 16 21:40:30.153941 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.153901 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:30.156634 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.156564 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bbd464946-dpq6k"] Jul 16 21:40:30.156860 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.156563 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 21:40:30.157127 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.156678 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7btkb\"" Jul 16 21:40:30.157785 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.157760 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8"] Jul 16 21:40:30.224304 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.224265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ecf63b0-0e83-46ad-ac13-548c487771b1-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-c6jq8\" (UID: \"0ecf63b0-0e83-46ad-ac13-548c487771b1\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:30.224617 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.224375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftts\" (UniqueName: \"kubernetes.io/projected/0ecf63b0-0e83-46ad-ac13-548c487771b1-kube-api-access-sftts\") pod \"monitoring-plugin-7b7b7bc8bf-c6jq8\" (UID: \"0ecf63b0-0e83-46ad-ac13-548c487771b1\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:30.224617 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.224435 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:30.288638 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.288601 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:30.326156 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.325932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ecf63b0-0e83-46ad-ac13-548c487771b1-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-c6jq8\" (UID: \"0ecf63b0-0e83-46ad-ac13-548c487771b1\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:30.326156 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.326047 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftts\" (UniqueName: \"kubernetes.io/projected/0ecf63b0-0e83-46ad-ac13-548c487771b1-kube-api-access-sftts\") pod \"monitoring-plugin-7b7b7bc8bf-c6jq8\" (UID: \"0ecf63b0-0e83-46ad-ac13-548c487771b1\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:30.328846 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.328810 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ecf63b0-0e83-46ad-ac13-548c487771b1-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-c6jq8\" (UID: \"0ecf63b0-0e83-46ad-ac13-548c487771b1\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:30.335383 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.335346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftts\" (UniqueName: \"kubernetes.io/projected/0ecf63b0-0e83-46ad-ac13-548c487771b1-kube-api-access-sftts\") pod \"monitoring-plugin-7b7b7bc8bf-c6jq8\" (UID: \"0ecf63b0-0e83-46ad-ac13-548c487771b1\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:30.469715 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.469626 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:30.533290 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.533256 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv"] Jul 16 21:40:30.538079 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.538049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.540742 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.540715 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 21:40:30.540742 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.540735 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 21:40:30.540969 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.540715 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 21:40:30.540969 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.540713 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 21:40:30.540969 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.540840 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 21:40:30.541217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.541196 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8lq9h\"" Jul 16 21:40:30.546729 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.546699 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 21:40:30.549937 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.549904 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv"] Jul 16 21:40:30.622593 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.622558 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8"] Jul 16 21:40:30.626578 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:30.626552 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecf63b0_0e83_46ad_ac13_548c487771b1.slice/crio-049820eb051d195ef8cc27f1376de0caf33fe9661d5b40fb460218131d5ebe17 WatchSource:0}: Error finding container 049820eb051d195ef8cc27f1376de0caf33fe9661d5b40fb460218131d5ebe17: Status 404 returned error can't find the container with id 049820eb051d195ef8cc27f1376de0caf33fe9661d5b40fb460218131d5ebe17 Jul 16 21:40:30.627920 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.627884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.628017 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.627932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qkk7\" (UniqueName: \"kubernetes.io/projected/816ddd8c-e681-4b27-84f3-7ad1881838f8-kube-api-access-5qkk7\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.628017 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.627965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-serving-certs-ca-bundle\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.628017 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.627984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-secret-telemeter-client\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.628017 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.628008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-metrics-client-ca\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.628325 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.628075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-telemeter-client-tls\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.628325 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.628125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.628325 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.628219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-federate-client-tls\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.729577 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.729448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-telemeter-client-tls\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.729577 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.729496 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730063 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.729705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-federate-client-tls\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730063 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.729811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730063 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.729843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qkk7\" (UniqueName: \"kubernetes.io/projected/816ddd8c-e681-4b27-84f3-7ad1881838f8-kube-api-access-5qkk7\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730063 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.729879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-serving-certs-ca-bundle\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730063 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.729907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-secret-telemeter-client\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730310 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.730125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-metrics-client-ca\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730639 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.730612 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-serving-certs-ca-bundle\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730851 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.730828 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.730931 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.730905 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/816ddd8c-e681-4b27-84f3-7ad1881838f8-metrics-client-ca\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.732275 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.732243 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-federate-client-tls\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.732478 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.732460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.732734 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.732710 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-secret-telemeter-client\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.732734 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.732723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/816ddd8c-e681-4b27-84f3-7ad1881838f8-telemeter-client-tls\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.737668 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.737639 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qkk7\" (UniqueName: \"kubernetes.io/projected/816ddd8c-e681-4b27-84f3-7ad1881838f8-kube-api-access-5qkk7\") pod \"telemeter-client-75cccb9b5b-6cjgv\" (UID: \"816ddd8c-e681-4b27-84f3-7ad1881838f8\") " pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.851063 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.851031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" Jul 16 21:40:30.981945 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.981859 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerID="3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b" exitCode=0 Jul 16 21:40:30.982100 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.981963 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerDied","Data":"3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b"} Jul 16 21:40:30.985571 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.985403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" event={"ID":"362547e8-d75f-4861-b842-d5aec9f80ba9","Type":"ContainerStarted","Data":"4e170fbd30b8122360ed92d5f2c45a2c81bb83b93e56d27848b242e9c07e0f94"} Jul 16 21:40:30.985571 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.985476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" event={"ID":"362547e8-d75f-4861-b842-d5aec9f80ba9","Type":"ContainerStarted","Data":"b842bd1312b37b33c229b83c1e8a1c59b9b5cec0973dac388ef7560c4085a3f2"} Jul 16 21:40:30.988722 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.988686 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d9jbv" event={"ID":"d6e14b29-ad65-44ed-8119-8333506ad460","Type":"ContainerStarted","Data":"1645b416e632ed5056c8e8f29760b41b2d4e88cbfe4c314a2b47018b9d138d0d"} Jul 16 21:40:30.990187 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.990156 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" event={"ID":"69485f8e-111b-4abe-8669-549bb5bb6fe5","Type":"ContainerStarted","Data":"f4949f0b42a04abce149722873346256ac0c5afe1417086005452505d1cbf5c2"} Jul 16 21:40:30.991417 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:30.991393 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" event={"ID":"0ecf63b0-0e83-46ad-ac13-548c487771b1","Type":"ContainerStarted","Data":"049820eb051d195ef8cc27f1376de0caf33fe9661d5b40fb460218131d5ebe17"} Jul 16 21:40:31.010355 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.010323 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv"] Jul 16 21:40:31.013110 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:31.013082 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816ddd8c_e681_4b27_84f3_7ad1881838f8.slice/crio-bf379f2c78b3bc1c95711dedba37316e6237e85911e2a5949417e700572d4cd4 WatchSource:0}: Error finding container bf379f2c78b3bc1c95711dedba37316e6237e85911e2a5949417e700572d4cd4: Status 404 returned error can't find the container with id bf379f2c78b3bc1c95711dedba37316e6237e85911e2a5949417e700572d4cd4 Jul 16 21:40:31.023820 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.023768 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d9jbv" podStartSLOduration=5.077822003 podStartE2EDuration="6.02374985s" totalImagesPullingTime="864.732713ms" totalInitContainerRuntime="81.195134ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:31.022114353 +0000 UTC m=+78.064338591" watchObservedRunningTime="2026-07-16 21:40:31.02374985 +0000 UTC m=+78.065974088" Jul 16 21:40:31.041242 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.041189 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-hqjsh" podStartSLOduration=4.08902841 podStartE2EDuration="6.041169271s" totalImagesPullingTime="1.952140861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:31.039797029 +0000 UTC m=+78.082021268" watchObservedRunningTime="2026-07-16 21:40:31.041169271 +0000 UTC m=+78.083393510" Jul 16 21:40:31.379765 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.379712 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b8669d59-gvc6x"] Jul 16 21:40:31.411175 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.411108 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8678b4966f-69vn4"] Jul 16 21:40:31.418082 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.418049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.430111 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.428505 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8678b4966f-69vn4"] Jul 16 21:40:31.537762 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.537723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-trusted-ca-bundle\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.537964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.537778 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-serving-cert\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.537964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.537809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-oauth-config\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.537964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.537845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-console-config\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.537964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.537908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-service-ca\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.537964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.537943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-oauth-serving-cert\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.538243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.537980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kp8\" (UniqueName: \"kubernetes.io/projected/cd809772-245c-4547-a58b-178fc539959e-kube-api-access-77kp8\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.574881 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.574786 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:31.579890 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.579799 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:40:31.585070 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.585037 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.588600 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.588567 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 21:40:31.588760 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.588655 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 21:40:31.588924 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.588905 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 21:40:31.589015 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.588934 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 21:40:31.589225 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.589082 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 21:40:31.589225 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.589220 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 21:40:31.589413 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.589367 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 21:40:31.589592 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.589416 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q7x7k\"" Jul 16 21:40:31.589757 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.589699 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1pkqkt4vsgplp\"" Jul 16 21:40:31.589757 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.589741 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 21:40:31.590824 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.589983 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 21:40:31.590824 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.590065 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 21:40:31.590824 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.590134 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 21:40:31.590824 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.590172 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 21:40:31.590824 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.589987 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 21:40:31.593385 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.593224 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 21:40:31.606589 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.605878 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:40:31.639449 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639449 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639449 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639708 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639708 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-config\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639708 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639708 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639662 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639708 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639909 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639714 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639909 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639909 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639909 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.639909 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.640073 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-trusted-ca-bundle\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.640073 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.639978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-serving-cert\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.640073 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-oauth-config\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.640073 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-console-config\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.640073 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.640243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-service-ca\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.640243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640149 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.640243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-oauth-serving-cert\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.640243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.640373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kp8\" (UniqueName: \"kubernetes.io/projected/cd809772-245c-4547-a58b-178fc539959e-kube-api-access-77kp8\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.640373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.640373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640296 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch27q\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-kube-api-access-ch27q\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.640373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640323 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.640373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.640873 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.640846 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-trusted-ca-bundle\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.641187 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.641164 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-oauth-serving-cert\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.641502 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.641482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-service-ca\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.641789 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.641764 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-console-config\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.644007 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.643975 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-oauth-config\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.644668 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.644644 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-serving-cert\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.656211 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.656101 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kp8\" (UniqueName: \"kubernetes.io/projected/cd809772-245c-4547-a58b-178fc539959e-kube-api-access-77kp8\") pod \"console-8678b4966f-69vn4\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.734887 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.734846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:31.742557 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.742754 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.742754 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.742754 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch27q\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-kube-api-access-ch27q\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.742991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-config\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743194 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743232 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.743397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.743325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.745088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.744044 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.745088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.744340 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.745088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.744368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.745361 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.745172 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.747288 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.747014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.748997 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.747664 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.748997 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.747014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.748997 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.748318 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.752277 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.751777 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.752277 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.752194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.752277 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.752211 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.752606 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.752364 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.753167 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.752767 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.753167 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.753132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.753737 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.753437 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.753737 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.753700 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.754035 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.754013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.754784 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.754762 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-config\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.755199 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.755148 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.765444 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.765372 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch27q\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-kube-api-access-ch27q\") pod \"prometheus-k8s-0\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:31.901084 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:31.900912 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:32.005940 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:32.005889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" event={"ID":"816ddd8c-e681-4b27-84f3-7ad1881838f8","Type":"ContainerStarted","Data":"bf379f2c78b3bc1c95711dedba37316e6237e85911e2a5949417e700572d4cd4"} Jul 16 21:40:32.207432 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:32.207235 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:32.268484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:32.268374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:32.295589 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:32.295554 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:32.295757 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:32.295625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:32.302544 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:32.302495 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:32.333177 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:32.333120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:32.395461 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:32.395422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:33.020538 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:33.020478 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:40:33.600328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:33.600291 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:33.701643 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:33.701592 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:34.501713 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:34.501683 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8678b4966f-69vn4"] Jul 16 21:40:34.517805 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:40:34.517746 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd809772_245c_4547_a58b_178fc539959e.slice/crio-1a175923f1b94f466c85a9417abf098df33efb733105bcf906f485b301a317ad WatchSource:0}: Error finding container 1a175923f1b94f466c85a9417abf098df33efb733105bcf906f485b301a317ad: Status 404 returned error can't find the container with id 1a175923f1b94f466c85a9417abf098df33efb733105bcf906f485b301a317ad Jul 16 21:40:34.547574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:34.546301 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:40:34.733965 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:34.733848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:34.796182 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:34.796151 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:35.034375 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.034343 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerStarted","Data":"be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2"} Jul 16 21:40:35.034566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.034451 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerStarted","Data":"3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c"} Jul 16 21:40:35.034566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.034485 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerStarted","Data":"e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41"} Jul 16 21:40:35.034566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.034510 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerStarted","Data":"c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce"} Jul 16 21:40:35.034715 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.034571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerStarted","Data":"5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a"} Jul 16 21:40:35.036488 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.036458 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" event={"ID":"816ddd8c-e681-4b27-84f3-7ad1881838f8","Type":"ContainerStarted","Data":"db28828b54752c52005d3565b78302740fdb621e02ee4aed6410af3424eec86f"} Jul 16 21:40:35.036650 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.036535 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" event={"ID":"816ddd8c-e681-4b27-84f3-7ad1881838f8","Type":"ContainerStarted","Data":"e8b6ede806d3cc457b30929994924abc36d396d852e92a76b226e30834386ed1"} Jul 16 21:40:35.036650 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.036568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" event={"ID":"816ddd8c-e681-4b27-84f3-7ad1881838f8","Type":"ContainerStarted","Data":"9fb6fe46eb8d679caafad764e9cb6f4c5e3e74ae7d71afd40212053c6a0bc4be"} Jul 16 21:40:35.038609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.038585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" event={"ID":"1357ede3-e5c3-403f-806f-258928451826","Type":"ContainerStarted","Data":"3954a4d8a595c4c6464b55f82acd79165083f1be2ce0b262a8c8308953f4241c"} Jul 16 21:40:35.038706 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.038649 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" event={"ID":"1357ede3-e5c3-403f-806f-258928451826","Type":"ContainerStarted","Data":"931323ef6d135a45ddaa9fdf2631a375c69eff22504b54c207d6d9361edfc123"} Jul 16 21:40:35.038706 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.038692 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" event={"ID":"1357ede3-e5c3-403f-806f-258928451826","Type":"ContainerStarted","Data":"7298a48581df2989fa9a238c815286ce57ee63f127b191c13c721f4c7338c64a"} Jul 16 21:40:35.040223 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.040151 2573 generic.go:356] "Generic (PLEG): container finished" podID="2630e476-43b2-4db8-be71-62971d5e164c" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" exitCode=0 Jul 16 21:40:35.040321 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.040234 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerDied","Data":"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2"} Jul 16 21:40:35.040321 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.040306 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerStarted","Data":"f6b47631a3a8b53c93061d39aff3a23b521ee7db0ffb8eb712d2a21251d46f03"} Jul 16 21:40:35.041819 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.041793 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" event={"ID":"0ecf63b0-0e83-46ad-ac13-548c487771b1","Type":"ContainerStarted","Data":"439b09f1efb2abc69a78a1e56f6db69f2c35ad89921219f5da5a4175ca8dd168"} Jul 16 21:40:35.042068 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.042021 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:35.043623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.043592 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8678b4966f-69vn4" event={"ID":"cd809772-245c-4547-a58b-178fc539959e","Type":"ContainerStarted","Data":"9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b"} Jul 16 21:40:35.043732 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.043676 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8678b4966f-69vn4" event={"ID":"cd809772-245c-4547-a58b-178fc539959e","Type":"ContainerStarted","Data":"1a175923f1b94f466c85a9417abf098df33efb733105bcf906f485b301a317ad"} Jul 16 21:40:35.045219 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.045199 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" event={"ID":"69485f8e-111b-4abe-8669-549bb5bb6fe5","Type":"ContainerStarted","Data":"9590840a48f836584b15c232dda1fb46ab2cda4cd81c551117e7f9e41f8aee34"} Jul 16 21:40:35.047387 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.047369 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" Jul 16 21:40:35.058781 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.058719 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75cccb9b5b-6cjgv" podStartSLOduration=1.769431257 podStartE2EDuration="5.058700541s" totalImagesPullingTime="3.289269284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:35.057608737 +0000 UTC m=+82.099832978" watchObservedRunningTime="2026-07-16 21:40:35.058700541 +0000 UTC m=+82.100924780" Jul 16 21:40:35.078328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.078270 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8678b4966f-69vn4" podStartSLOduration=4.078251553 podStartE2EDuration="4.078251553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:35.076561543 +0000 UTC m=+82.118785772" watchObservedRunningTime="2026-07-16 21:40:35.078251553 +0000 UTC m=+82.120475791" Jul 16 21:40:35.097159 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.097106 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" podStartSLOduration=1.961578649 podStartE2EDuration="6.097088216s" totalImagesPullingTime="4.135509567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:35.096512647 +0000 UTC m=+82.138736877" watchObservedRunningTime="2026-07-16 21:40:35.097088216 +0000 UTC m=+82.139312454" Jul 16 21:40:35.116320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.116264 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-c6jq8" podStartSLOduration=1.444569424 podStartE2EDuration="5.116245068s" totalImagesPullingTime="3.671675644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:35.115569903 +0000 UTC m=+82.157794145" watchObservedRunningTime="2026-07-16 21:40:35.116245068 +0000 UTC m=+82.158469305" Jul 16 21:40:35.165689 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.165657 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:35.223578 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.223542 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:35.681287 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.681253 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:35.783220 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.783186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:35.860124 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.860087 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8678b4966f-69vn4"] Jul 16 21:40:35.938225 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.937930 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:35.997582 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:35.997540 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:37.063896 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:37.063859 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerStarted","Data":"01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23"} Jul 16 21:40:37.068147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:37.067454 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" event={"ID":"1357ede3-e5c3-403f-806f-258928451826","Type":"ContainerStarted","Data":"a1fd0b188d7d1ac310a2e5881d49cd268827e8a813591ff8952ecb62b0eb2264"} Jul 16 21:40:37.068147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:37.067556 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" event={"ID":"1357ede3-e5c3-403f-806f-258928451826","Type":"ContainerStarted","Data":"c9fb4d198002a4274ca70bd7c78aff80c9e5f00e3ada0c99bf5fcd61ee74f3b4"} Jul 16 21:40:37.068147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:37.067599 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" event={"ID":"1357ede3-e5c3-403f-806f-258928451826","Type":"ContainerStarted","Data":"683a44de18a74705830c51a721125b14cabaaf712adf76ee9d1ff9c8bac195be"} Jul 16 21:40:37.095690 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:37.095631 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.471580286 podStartE2EDuration="11.095611762s" totalImagesPullingTime="6.594239085s" totalInitContainerRuntime="29.792391ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:37.093871124 +0000 UTC m=+84.136095362" watchObservedRunningTime="2026-07-16 21:40:37.095611762 +0000 UTC m=+84.137836015" Jul 16 21:40:37.133322 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:37.133259 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" podStartSLOduration=2.260631911 podStartE2EDuration="9.133242396s" totalImagesPullingTime="6.872610485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:37.130645208 +0000 UTC m=+84.172869458" watchObservedRunningTime="2026-07-16 21:40:37.133242396 +0000 UTC m=+84.175466634" Jul 16 21:40:38.073436 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:38.073398 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:39.086371 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:39.086342 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bd44f97fc-mvdsv" Jul 16 21:40:39.220365 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:39.220326 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:39.321226 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:39.320714 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:40.086478 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:40.086443 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerStarted","Data":"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf"} Jul 16 21:40:40.086862 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:40.086515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerStarted","Data":"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8"} Jul 16 21:40:40.086862 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:40.086570 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerStarted","Data":"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814"} Jul 16 21:40:40.086862 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:40.086597 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerStarted","Data":"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd"} Jul 16 21:40:40.086862 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:40.086621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerStarted","Data":"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c"} Jul 16 21:40:40.086862 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:40.086646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerStarted","Data":"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b"} Jul 16 21:40:40.119402 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:40.119350 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.099686676 podStartE2EDuration="9.119333048s" totalImagesPullingTime="3.990507093s" totalInitContainerRuntime="29.139279ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:40.116400905 +0000 UTC m=+87.158625180" watchObservedRunningTime="2026-07-16 21:40:40.119333048 +0000 UTC m=+87.161557285" Jul 16 21:40:41.735423 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:41.735386 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:40:41.901637 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:41.901603 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:40:45.001373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:45.001334 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:45.100555 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:45.100499 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:46.358844 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:46.358811 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f648488-qr5ht"] Jul 16 21:40:46.481305 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:46.481267 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:46.582328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:46.582277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:46.685030 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:46.684944 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:49.744707 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:49.744670 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:49.848374 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:49.848336 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:49.948091 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:49.948050 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:49.948286 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:49.948122 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:40:56.408802 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:56.408736 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55b8669d59-gvc6x" podUID="b33ec7ed-281a-4dff-be63-3f6e7453a984" containerName="console" containerID="cri-o://1ee6192f4579d815136be37f9d535dd36bb368326464a62cadc65dcfad3ba714" gracePeriod=15 Jul 16 21:40:57.164371 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.164336 2573 generic.go:356] "Generic (PLEG): container finished" podID="b33ec7ed-281a-4dff-be63-3f6e7453a984" containerID="1ee6192f4579d815136be37f9d535dd36bb368326464a62cadc65dcfad3ba714" exitCode=0 Jul 16 21:40:57.164587 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.164408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b8669d59-gvc6x" event={"ID":"b33ec7ed-281a-4dff-be63-3f6e7453a984","Type":"ContainerDied","Data":"1ee6192f4579d815136be37f9d535dd36bb368326464a62cadc65dcfad3ba714"} Jul 16 21:40:57.164587 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.164457 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b8669d59-gvc6x" event={"ID":"b33ec7ed-281a-4dff-be63-3f6e7453a984","Type":"ContainerDied","Data":"874bf5c3d4bba2942bbc01ae1a609b397cdf51022ac13616422578a2489a109e"} Jul 16 21:40:57.164587 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.164472 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="874bf5c3d4bba2942bbc01ae1a609b397cdf51022ac13616422578a2489a109e" Jul 16 21:40:57.193645 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.193603 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:57.309130 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309098 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-serving-cert\") pod \"b33ec7ed-281a-4dff-be63-3f6e7453a984\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " Jul 16 21:40:57.309296 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309137 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-service-ca\") pod \"b33ec7ed-281a-4dff-be63-3f6e7453a984\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " Jul 16 21:40:57.309296 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309190 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-oauth-config\") pod \"b33ec7ed-281a-4dff-be63-3f6e7453a984\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " Jul 16 21:40:57.309296 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309240 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-config\") pod \"b33ec7ed-281a-4dff-be63-3f6e7453a984\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " Jul 16 21:40:57.309296 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309276 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7z95\" (UniqueName: \"kubernetes.io/projected/b33ec7ed-281a-4dff-be63-3f6e7453a984-kube-api-access-x7z95\") pod \"b33ec7ed-281a-4dff-be63-3f6e7453a984\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " Jul 16 21:40:57.309450 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309325 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-oauth-serving-cert\") pod \"b33ec7ed-281a-4dff-be63-3f6e7453a984\" (UID: \"b33ec7ed-281a-4dff-be63-3f6e7453a984\") " Jul 16 21:40:57.309751 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309724 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-config" (OuterVolumeSpecName: "console-config") pod "b33ec7ed-281a-4dff-be63-3f6e7453a984" (UID: "b33ec7ed-281a-4dff-be63-3f6e7453a984"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:40:57.309866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309743 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-service-ca" (OuterVolumeSpecName: "service-ca") pod "b33ec7ed-281a-4dff-be63-3f6e7453a984" (UID: "b33ec7ed-281a-4dff-be63-3f6e7453a984"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:40:57.309866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.309803 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b33ec7ed-281a-4dff-be63-3f6e7453a984" (UID: "b33ec7ed-281a-4dff-be63-3f6e7453a984"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:40:57.311681 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.311650 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b33ec7ed-281a-4dff-be63-3f6e7453a984" (UID: "b33ec7ed-281a-4dff-be63-3f6e7453a984"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:40:57.311789 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.311696 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33ec7ed-281a-4dff-be63-3f6e7453a984-kube-api-access-x7z95" (OuterVolumeSpecName: "kube-api-access-x7z95") pod "b33ec7ed-281a-4dff-be63-3f6e7453a984" (UID: "b33ec7ed-281a-4dff-be63-3f6e7453a984"). InnerVolumeSpecName "kube-api-access-x7z95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:40:57.311789 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.311696 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b33ec7ed-281a-4dff-be63-3f6e7453a984" (UID: "b33ec7ed-281a-4dff-be63-3f6e7453a984"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:40:57.410876 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.410833 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-serving-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:40:57.410876 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.410871 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:40:57.410876 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.410887 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-oauth-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:40:57.411336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.410899 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-console-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:40:57.411336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.410911 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7z95\" (UniqueName: \"kubernetes.io/projected/b33ec7ed-281a-4dff-be63-3f6e7453a984-kube-api-access-x7z95\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:40:57.411336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:57.410923 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b33ec7ed-281a-4dff-be63-3f6e7453a984-oauth-serving-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:40:58.168806 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:58.168770 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b8669d59-gvc6x" Jul 16 21:40:58.185893 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:58.185856 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b8669d59-gvc6x"] Jul 16 21:40:58.189713 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:58.189684 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55b8669d59-gvc6x"] Jul 16 21:40:58.960484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:58.960452 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4qbsw" Jul 16 21:40:59.323925 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:59.323881 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:59.381282 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:59.381248 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:40:59.540442 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:40:59.540402 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33ec7ed-281a-4dff-be63-3f6e7453a984" path="/var/lib/kubelet/pods/b33ec7ed-281a-4dff-be63-3f6e7453a984/volumes" Jul 16 21:41:02.093453 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.093405 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8678b4966f-69vn4" podUID="cd809772-245c-4547-a58b-178fc539959e" containerName="console" containerID="cri-o://9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b" gracePeriod=15 Jul 16 21:41:02.376358 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.376278 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:41:02.453262 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453228 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-console-config\") pod \"cd809772-245c-4547-a58b-178fc539959e\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " Jul 16 21:41:02.453458 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453287 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-oauth-serving-cert\") pod \"cd809772-245c-4547-a58b-178fc539959e\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " Jul 16 21:41:02.453458 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453319 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77kp8\" (UniqueName: \"kubernetes.io/projected/cd809772-245c-4547-a58b-178fc539959e-kube-api-access-77kp8\") pod \"cd809772-245c-4547-a58b-178fc539959e\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " Jul 16 21:41:02.453458 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453343 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-serving-cert\") pod \"cd809772-245c-4547-a58b-178fc539959e\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " Jul 16 21:41:02.453458 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453365 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-service-ca\") pod \"cd809772-245c-4547-a58b-178fc539959e\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " Jul 16 21:41:02.453458 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453409 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-trusted-ca-bundle\") pod \"cd809772-245c-4547-a58b-178fc539959e\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " Jul 16 21:41:02.453458 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453442 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-oauth-config\") pod \"cd809772-245c-4547-a58b-178fc539959e\" (UID: \"cd809772-245c-4547-a58b-178fc539959e\") " Jul 16 21:41:02.453830 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cd809772-245c-4547-a58b-178fc539959e" (UID: "cd809772-245c-4547-a58b-178fc539959e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:02.453947 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453867 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-service-ca" (OuterVolumeSpecName: "service-ca") pod "cd809772-245c-4547-a58b-178fc539959e" (UID: "cd809772-245c-4547-a58b-178fc539959e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:02.453993 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.453941 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cd809772-245c-4547-a58b-178fc539959e" (UID: "cd809772-245c-4547-a58b-178fc539959e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:02.455626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.455590 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cd809772-245c-4547-a58b-178fc539959e" (UID: "cd809772-245c-4547-a58b-178fc539959e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:02.455626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.455615 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cd809772-245c-4547-a58b-178fc539959e" (UID: "cd809772-245c-4547-a58b-178fc539959e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:02.455784 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.455694 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd809772-245c-4547-a58b-178fc539959e-kube-api-access-77kp8" (OuterVolumeSpecName: "kube-api-access-77kp8") pod "cd809772-245c-4547-a58b-178fc539959e" (UID: "cd809772-245c-4547-a58b-178fc539959e"). InnerVolumeSpecName "kube-api-access-77kp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:41:02.460747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.460722 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-console-config" (OuterVolumeSpecName: "console-config") pod "cd809772-245c-4547-a58b-178fc539959e" (UID: "cd809772-245c-4547-a58b-178fc539959e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:02.554419 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.554370 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-console-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:02.554419 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.554411 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-oauth-serving-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:02.554419 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.554421 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77kp8\" (UniqueName: \"kubernetes.io/projected/cd809772-245c-4547-a58b-178fc539959e-kube-api-access-77kp8\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:02.554419 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.554430 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-serving-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:02.554719 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.554439 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:02.554719 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.554447 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd809772-245c-4547-a58b-178fc539959e-trusted-ca-bundle\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:02.554719 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:02.554455 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd809772-245c-4547-a58b-178fc539959e-console-oauth-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:03.196194 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.196154 2573 generic.go:356] "Generic (PLEG): container finished" podID="cd809772-245c-4547-a58b-178fc539959e" containerID="9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b" exitCode=0 Jul 16 21:41:03.196652 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.196210 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8678b4966f-69vn4" Jul 16 21:41:03.196652 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.196224 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8678b4966f-69vn4" event={"ID":"cd809772-245c-4547-a58b-178fc539959e","Type":"ContainerDied","Data":"9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b"} Jul 16 21:41:03.196652 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.196274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8678b4966f-69vn4" event={"ID":"cd809772-245c-4547-a58b-178fc539959e","Type":"ContainerDied","Data":"1a175923f1b94f466c85a9417abf098df33efb733105bcf906f485b301a317ad"} Jul 16 21:41:03.196652 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.196300 2573 scope.go:117] "RemoveContainer" containerID="9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b" Jul 16 21:41:03.205290 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.205262 2573 scope.go:117] "RemoveContainer" containerID="9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b" Jul 16 21:41:03.205628 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:03.205607 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b\": container with ID starting with 9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b not found: ID does not exist" containerID="9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b" Jul 16 21:41:03.205689 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.205640 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b"} err="failed to get container status \"9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b\": rpc error: code = NotFound desc = could not find container \"9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b\": container with ID starting with 9acd2580e19ae4053d99b545bd3f8bb6b9358ff645608eb822b683f589bdd49b not found: ID does not exist" Jul 16 21:41:03.216836 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.216800 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8678b4966f-69vn4"] Jul 16 21:41:03.221421 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.221392 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8678b4966f-69vn4"] Jul 16 21:41:03.540680 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:03.540598 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd809772-245c-4547-a58b-178fc539959e" path="/var/lib/kubelet/pods/cd809772-245c-4547-a58b-178fc539959e/volumes" Jul 16 21:41:04.745871 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:04.745827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:04.800150 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:04.800109 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:09.953731 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:09.953702 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:41:09.957820 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:09.957793 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bbd464946-dpq6k" Jul 16 21:41:11.382680 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.382638 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55f648488-qr5ht" podUID="0ab77e64-ef70-47c0-abef-0cf2954cdd3d" containerName="console" containerID="cri-o://eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84" gracePeriod=15 Jul 16 21:41:11.624442 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.624412 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:41:11.693369 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.693278 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:11.727295 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727263 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-trusted-ca-bundle\") pod \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " Jul 16 21:41:11.727416 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727314 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-config\") pod \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " Jul 16 21:41:11.727416 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727340 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-service-ca\") pod \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " Jul 16 21:41:11.727416 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727360 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-serving-cert\") pod \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " Jul 16 21:41:11.727601 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727482 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d2vq\" (UniqueName: \"kubernetes.io/projected/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-kube-api-access-6d2vq\") pod \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " Jul 16 21:41:11.727601 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727590 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-oauth-config\") pod \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " Jul 16 21:41:11.727704 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727638 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-oauth-serving-cert\") pod \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\" (UID: \"0ab77e64-ef70-47c0-abef-0cf2954cdd3d\") " Jul 16 21:41:11.727780 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727753 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-config" (OuterVolumeSpecName: "console-config") pod "0ab77e64-ef70-47c0-abef-0cf2954cdd3d" (UID: "0ab77e64-ef70-47c0-abef-0cf2954cdd3d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:11.727780 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727771 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0ab77e64-ef70-47c0-abef-0cf2954cdd3d" (UID: "0ab77e64-ef70-47c0-abef-0cf2954cdd3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:11.727885 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727816 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-service-ca" (OuterVolumeSpecName: "service-ca") pod "0ab77e64-ef70-47c0-abef-0cf2954cdd3d" (UID: "0ab77e64-ef70-47c0-abef-0cf2954cdd3d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:11.727942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727924 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-trusted-ca-bundle\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:11.727989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727944 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:11.727989 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.727959 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:11.728107 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.728090 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0ab77e64-ef70-47c0-abef-0cf2954cdd3d" (UID: "0ab77e64-ef70-47c0-abef-0cf2954cdd3d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:11.729788 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.729761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0ab77e64-ef70-47c0-abef-0cf2954cdd3d" (UID: "0ab77e64-ef70-47c0-abef-0cf2954cdd3d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:11.730145 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.730118 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-kube-api-access-6d2vq" (OuterVolumeSpecName: "kube-api-access-6d2vq") pod "0ab77e64-ef70-47c0-abef-0cf2954cdd3d" (UID: "0ab77e64-ef70-47c0-abef-0cf2954cdd3d"). InnerVolumeSpecName "kube-api-access-6d2vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:41:11.730145 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.730136 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0ab77e64-ef70-47c0-abef-0cf2954cdd3d" (UID: "0ab77e64-ef70-47c0-abef-0cf2954cdd3d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:11.798152 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.798120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:11.829364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.829330 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-serving-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:11.829364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.829358 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2vq\" (UniqueName: \"kubernetes.io/projected/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-kube-api-access-6d2vq\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:11.829364 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.829370 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-console-oauth-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:11.829607 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:11.829380 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab77e64-ef70-47c0-abef-0cf2954cdd3d-oauth-serving-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:12.242581 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.242548 2573 generic.go:356] "Generic (PLEG): container finished" podID="0ab77e64-ef70-47c0-abef-0cf2954cdd3d" containerID="eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84" exitCode=0 Jul 16 21:41:12.242581 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.242587 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f648488-qr5ht" Jul 16 21:41:12.242866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.242622 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f648488-qr5ht" event={"ID":"0ab77e64-ef70-47c0-abef-0cf2954cdd3d","Type":"ContainerDied","Data":"eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84"} Jul 16 21:41:12.242866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.242690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f648488-qr5ht" event={"ID":"0ab77e64-ef70-47c0-abef-0cf2954cdd3d","Type":"ContainerDied","Data":"b9090ea728151994ab569b160c8ab6a1f0b2b4223747a18833b7f0cace1f772e"} Jul 16 21:41:12.242866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.242719 2573 scope.go:117] "RemoveContainer" containerID="eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84" Jul 16 21:41:12.251640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.251618 2573 scope.go:117] "RemoveContainer" containerID="eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84" Jul 16 21:41:12.251950 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:12.251929 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84\": container with ID starting with eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84 not found: ID does not exist" containerID="eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84" Jul 16 21:41:12.252000 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.251959 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84"} err="failed to get container status \"eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84\": rpc error: code = NotFound desc = could not find container \"eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84\": container with ID starting with eb0d2f6885373e038e125456834b8afc691116a41fe1cb35d3a447cc43dd5e84 not found: ID does not exist" Jul 16 21:41:12.262701 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.262665 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f648488-qr5ht"] Jul 16 21:41:12.266367 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.266340 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55f648488-qr5ht"] Jul 16 21:41:12.613910 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:12.613866 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:13.540175 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:13.540138 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab77e64-ef70-47c0-abef-0cf2954cdd3d" path="/var/lib/kubelet/pods/0ab77e64-ef70-47c0-abef-0cf2954cdd3d/volumes" Jul 16 21:41:31.222740 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:31.222699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:31.901231 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:31.901189 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:31.920733 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:31.920703 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:32.337460 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:32.337432 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:41.428656 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:41.428620 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:41.487397 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:41.487346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:43.249683 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:43.249644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:45.702418 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:45.702378 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:41:45.702927 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:45.702697 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="alertmanager" containerID="cri-o://5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a" gracePeriod=120 Jul 16 21:41:45.702927 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:45.702819 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy-web" containerID="cri-o://e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41" gracePeriod=120 Jul 16 21:41:45.703439 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:45.702845 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy" containerID="cri-o://3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c" gracePeriod=120 Jul 16 21:41:45.703439 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:45.702849 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="config-reloader" containerID="cri-o://c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce" gracePeriod=120 Jul 16 21:41:45.703439 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:45.702821 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="prom-label-proxy" containerID="cri-o://01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23" gracePeriod=120 Jul 16 21:41:45.703439 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:45.702789 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy-metric" containerID="cri-o://be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2" gracePeriod=120 Jul 16 21:41:46.385343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385311 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerID="01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23" exitCode=0 Jul 16 21:41:46.385343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385336 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerID="be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2" exitCode=0 Jul 16 21:41:46.385343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385343 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerID="3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c" exitCode=0 Jul 16 21:41:46.385343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385348 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerID="c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce" exitCode=0 Jul 16 21:41:46.385343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385353 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerID="5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a" exitCode=0 Jul 16 21:41:46.385661 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385385 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerDied","Data":"01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23"} Jul 16 21:41:46.385661 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385447 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerDied","Data":"be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2"} Jul 16 21:41:46.385661 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385474 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerDied","Data":"3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c"} Jul 16 21:41:46.385661 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385499 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerDied","Data":"c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce"} Jul 16 21:41:46.385661 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.385541 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerDied","Data":"5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a"} Jul 16 21:41:46.951150 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:46.951122 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.047730 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047701 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52z4s\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-kube-api-access-52z4s\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.047908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047743 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-web\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.047908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047785 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-main-tls\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.047908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047821 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-web-config\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.047908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047842 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-cluster-tls-config\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.047908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047871 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.047908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047906 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-tls-assets\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.048218 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047956 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-main-db\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.048218 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.047989 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.048330 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.048280 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:41:47.048398 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.048359 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:47.049252 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.048541 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-volume\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.049252 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.048594 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.049252 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.048624 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-out\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.049252 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.048668 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-metrics-client-ca\") pod \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\" (UID: \"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b\") " Jul 16 21:41:47.049252 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.048949 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-main-db\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.049252 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.048969 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.049646 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.049352 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:47.051135 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.051107 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:47.052179 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.052141 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-kube-api-access-52z4s" (OuterVolumeSpecName: "kube-api-access-52z4s") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "kube-api-access-52z4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:41:47.052450 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.052400 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:41:47.052450 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.052410 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:47.052702 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.052676 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-volume" (OuterVolumeSpecName: "config-volume") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:47.052785 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.052733 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:47.052918 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.052891 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:47.053028 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.053006 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-out" (OuterVolumeSpecName: "config-out") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:41:47.055974 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.055945 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:47.063042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.063015 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-web-config" (OuterVolumeSpecName: "web-config") pod "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" (UID: "cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:47.150128 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150097 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-web-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150128 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150122 2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-cluster-tls-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150128 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150133 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150357 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150143 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-tls-assets\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150357 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150151 2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150357 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150161 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150357 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150169 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-config-out\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150357 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150177 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-metrics-client-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150357 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150186 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52z4s\" (UniqueName: \"kubernetes.io/projected/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-kube-api-access-52z4s\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150357 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150195 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.150357 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.150203 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b-secret-alertmanager-main-tls\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:47.391963 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.391870 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerID="e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41" exitCode=0 Jul 16 21:41:47.391963 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.391955 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.391963 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.391949 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerDied","Data":"e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41"} Jul 16 21:41:47.392233 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.392037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b","Type":"ContainerDied","Data":"33a1195252c7fa98919b5fd7e43c84f2edffe33fa15a7d67360b747451b1f9db"} Jul 16 21:41:47.392233 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.392086 2573 scope.go:117] "RemoveContainer" containerID="01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23" Jul 16 21:41:47.400103 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.400085 2573 scope.go:117] "RemoveContainer" containerID="be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2" Jul 16 21:41:47.407217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.407201 2573 scope.go:117] "RemoveContainer" containerID="3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c" Jul 16 21:41:47.415087 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.415066 2573 scope.go:117] "RemoveContainer" containerID="e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41" Jul 16 21:41:47.416065 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.416044 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:41:47.419196 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.419170 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:41:47.422248 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.422228 2573 scope.go:117] "RemoveContainer" containerID="c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce" Jul 16 21:41:47.429676 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.429658 2573 scope.go:117] "RemoveContainer" containerID="5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a" Jul 16 21:41:47.437018 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.436997 2573 scope.go:117] "RemoveContainer" containerID="3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b" Jul 16 21:41:47.444192 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.444169 2573 scope.go:117] "RemoveContainer" containerID="01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23" Jul 16 21:41:47.444694 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:47.444672 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23\": container with ID starting with 01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23 not found: ID does not exist" containerID="01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23" Jul 16 21:41:47.444779 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.444701 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23"} err="failed to get container status \"01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23\": rpc error: code = NotFound desc = could not find container \"01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23\": container with ID starting with 01ff3c3a4c861d0bc74cf46430855d234678007c87a186271e666063602d5d23 not found: ID does not exist" Jul 16 21:41:47.444779 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.444721 2573 scope.go:117] "RemoveContainer" containerID="be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2" Jul 16 21:41:47.444966 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:47.444941 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2\": container with ID starting with be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2 not found: ID does not exist" containerID="be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2" Jul 16 21:41:47.445004 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.444972 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2"} err="failed to get container status \"be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2\": rpc error: code = NotFound desc = could not find container \"be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2\": container with ID starting with be77cc0eddc0380a87a4b503d743bbb305b99deab8775dd93a9b36373dff0fc2 not found: ID does not exist" Jul 16 21:41:47.445004 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.444986 2573 scope.go:117] "RemoveContainer" containerID="3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c" Jul 16 21:41:47.445220 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:47.445206 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c\": container with ID starting with 3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c not found: ID does not exist" containerID="3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c" Jul 16 21:41:47.445258 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.445223 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c"} err="failed to get container status \"3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c\": rpc error: code = NotFound desc = could not find container \"3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c\": container with ID starting with 3aa457c4297d6df377fff19651f2bd0edcbb17ece30881a011ad95f8f470568c not found: ID does not exist" Jul 16 21:41:47.445258 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.445243 2573 scope.go:117] "RemoveContainer" containerID="e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41" Jul 16 21:41:47.445502 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:47.445483 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41\": container with ID starting with e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41 not found: ID does not exist" containerID="e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41" Jul 16 21:41:47.445610 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.445510 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41"} err="failed to get container status \"e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41\": rpc error: code = NotFound desc = could not find container \"e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41\": container with ID starting with e60efa0c4d713900b58f2ad57c8e0d7e9b79e896487624551b5ce8765c162a41 not found: ID does not exist" Jul 16 21:41:47.445610 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.445554 2573 scope.go:117] "RemoveContainer" containerID="c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce" Jul 16 21:41:47.445796 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:47.445778 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce\": container with ID starting with c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce not found: ID does not exist" containerID="c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce" Jul 16 21:41:47.445859 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.445804 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce"} err="failed to get container status \"c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce\": rpc error: code = NotFound desc = could not find container \"c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce\": container with ID starting with c6d8dd794ef350a0609272cc2a29505446dfdf5ab3432dcdcbe53ce046d675ce not found: ID does not exist" Jul 16 21:41:47.445859 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.445823 2573 scope.go:117] "RemoveContainer" containerID="5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a" Jul 16 21:41:47.446017 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.445999 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:41:47.446066 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:47.446031 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a\": container with ID starting with 5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a not found: ID does not exist" containerID="5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a" Jul 16 21:41:47.446066 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446047 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a"} err="failed to get container status \"5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a\": rpc error: code = NotFound desc = could not find container \"5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a\": container with ID starting with 5b2ed485f95fde8ecaa5ab76eb0c3b775f3d34207f3b4b676279bf74e6f6c40a not found: ID does not exist" Jul 16 21:41:47.446066 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446059 2573 scope.go:117] "RemoveContainer" containerID="3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b" Jul 16 21:41:47.446289 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:47.446267 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b\": container with ID starting with 3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b not found: ID does not exist" containerID="3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b" Jul 16 21:41:47.446336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446288 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd809772-245c-4547-a58b-178fc539959e" containerName="console" Jul 16 21:41:47.446336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446298 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd809772-245c-4547-a58b-178fc539959e" containerName="console" Jul 16 21:41:47.446336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446311 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="init-config-reloader" Jul 16 21:41:47.446336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446317 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="init-config-reloader" Jul 16 21:41:47.446336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446329 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy-web" Jul 16 21:41:47.446336 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446334 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy-web" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446343 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab77e64-ef70-47c0-abef-0cf2954cdd3d" containerName="console" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446349 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab77e64-ef70-47c0-abef-0cf2954cdd3d" containerName="console" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446357 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="alertmanager" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446362 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="alertmanager" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446293 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b"} err="failed to get container status \"3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b\": rpc error: code = NotFound desc = could not find container \"3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b\": container with ID starting with 3e04587d6f37e55da932dcd2283afa087b8b7bf1686de0cac611f22afb36c69b not found: ID does not exist" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446370 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="prom-label-proxy" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446375 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="prom-label-proxy" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446386 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33ec7ed-281a-4dff-be63-3f6e7453a984" containerName="console" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446391 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33ec7ed-281a-4dff-be63-3f6e7453a984" containerName="console" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446400 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="config-reloader" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446405 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="config-reloader" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446411 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy-metric" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446416 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy-metric" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446425 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446431 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446490 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446497 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy-metric" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446503 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd809772-245c-4547-a58b-178fc539959e" containerName="console" Jul 16 21:41:47.446504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446508 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="prom-label-proxy" Jul 16 21:41:47.447300 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446514 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="kube-rbac-proxy-web" Jul 16 21:41:47.447300 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446546 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33ec7ed-281a-4dff-be63-3f6e7453a984" containerName="console" Jul 16 21:41:47.447300 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446552 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="alertmanager" Jul 16 21:41:47.447300 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446557 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" containerName="config-reloader" Jul 16 21:41:47.447300 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.446562 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab77e64-ef70-47c0-abef-0cf2954cdd3d" containerName="console" Jul 16 21:41:47.450228 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.450213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.454249 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454225 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 21:41:47.454370 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454237 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 21:41:47.454370 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454277 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 21:41:47.454370 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454350 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7mwbr\"" Jul 16 21:41:47.454558 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454349 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 21:41:47.454558 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454383 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 21:41:47.454643 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454629 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 21:41:47.454966 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454853 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 21:41:47.455072 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.454983 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 21:41:47.458121 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.458093 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 21:41:47.464926 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.464901 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:41:47.540506 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.540473 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b" path="/var/lib/kubelet/pods/cc2e7e76-cb3c-4141-b2d8-e6b33f8eef6b/volumes" Jul 16 21:41:47.555442 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/042787c0-eece-41ee-b1ad-12bcd75922f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555553 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042787c0-eece-41ee-b1ad-12bcd75922f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555553 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/042787c0-eece-41ee-b1ad-12bcd75922f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555553 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/042787c0-eece-41ee-b1ad-12bcd75922f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555553 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555706 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555608 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555706 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555706 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/042787c0-eece-41ee-b1ad-12bcd75922f4-config-out\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555812 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555812 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555739 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555812 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555763 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvvg\" (UniqueName: \"kubernetes.io/projected/042787c0-eece-41ee-b1ad-12bcd75922f4-kube-api-access-qcvvg\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555812 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-web-config\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.555812 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.555806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.656851 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.656851 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.656851 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/042787c0-eece-41ee-b1ad-12bcd75922f4-config-out\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.656851 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvvg\" (UniqueName: \"kubernetes.io/projected/042787c0-eece-41ee-b1ad-12bcd75922f4-kube-api-access-qcvvg\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-web-config\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/042787c0-eece-41ee-b1ad-12bcd75922f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.656980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042787c0-eece-41ee-b1ad-12bcd75922f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.657011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/042787c0-eece-41ee-b1ad-12bcd75922f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.657048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/042787c0-eece-41ee-b1ad-12bcd75922f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.657186 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.657071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.659930 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.658802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/042787c0-eece-41ee-b1ad-12bcd75922f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.659930 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.659739 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/042787c0-eece-41ee-b1ad-12bcd75922f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.660183 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.660082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.660183 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.660153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.661047 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.660574 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.661047 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.660997 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-web-config\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.665378 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.661293 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.665378 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.661396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/042787c0-eece-41ee-b1ad-12bcd75922f4-config-out\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.665378 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.661763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.665378 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.661961 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/042787c0-eece-41ee-b1ad-12bcd75922f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.665378 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.662447 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/042787c0-eece-41ee-b1ad-12bcd75922f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.666199 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.666177 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042787c0-eece-41ee-b1ad-12bcd75922f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.668088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.668068 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvvg\" (UniqueName: \"kubernetes.io/projected/042787c0-eece-41ee-b1ad-12bcd75922f4-kube-api-access-qcvvg\") pod \"alertmanager-main-0\" (UID: \"042787c0-eece-41ee-b1ad-12bcd75922f4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.760869 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.760810 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:41:47.899304 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:47.899280 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:41:47.901448 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:41:47.901413 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042787c0_eece_41ee_b1ad_12bcd75922f4.slice/crio-f89c43b93cfa62a16c9b1e878aff7263ec57161535d0a577c3b406981908f6b6 WatchSource:0}: Error finding container f89c43b93cfa62a16c9b1e878aff7263ec57161535d0a577c3b406981908f6b6: Status 404 returned error can't find the container with id f89c43b93cfa62a16c9b1e878aff7263ec57161535d0a577c3b406981908f6b6 Jul 16 21:41:48.397891 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:48.397856 2573 generic.go:356] "Generic (PLEG): container finished" podID="042787c0-eece-41ee-b1ad-12bcd75922f4" containerID="bb07c6d2b453cb2d6f937c48f46a5b6d8fa1b68d37f67ab20be70105efe93d70" exitCode=0 Jul 16 21:41:48.398343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:48.397938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"042787c0-eece-41ee-b1ad-12bcd75922f4","Type":"ContainerDied","Data":"bb07c6d2b453cb2d6f937c48f46a5b6d8fa1b68d37f67ab20be70105efe93d70"} Jul 16 21:41:48.398343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:48.398002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"042787c0-eece-41ee-b1ad-12bcd75922f4","Type":"ContainerStarted","Data":"f89c43b93cfa62a16c9b1e878aff7263ec57161535d0a577c3b406981908f6b6"} Jul 16 21:41:49.406815 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.406778 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"042787c0-eece-41ee-b1ad-12bcd75922f4","Type":"ContainerStarted","Data":"893a54f5c6bade4c2e79847df5c35974abf7a45f4597894c737edb86dcd02bda"} Jul 16 21:41:49.407207 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.406859 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"042787c0-eece-41ee-b1ad-12bcd75922f4","Type":"ContainerStarted","Data":"48fd76f827f5d46904b8a6e836caa8f3494fe0e3b498bc5d3e74d5a99d5a9e3c"} Jul 16 21:41:49.407207 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.406898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"042787c0-eece-41ee-b1ad-12bcd75922f4","Type":"ContainerStarted","Data":"0e491d44bead4093dbeb1ddfdbd40fbfa4d79123cd978d4c23a65387ddcfdcdd"} Jul 16 21:41:49.407207 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.406937 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"042787c0-eece-41ee-b1ad-12bcd75922f4","Type":"ContainerStarted","Data":"91c402c8d9e1340349b8abb9fbe3143d2b866dd0cc0f222de7dbaf3ba8c03b69"} Jul 16 21:41:49.407207 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.406980 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"042787c0-eece-41ee-b1ad-12bcd75922f4","Type":"ContainerStarted","Data":"2af2f1af6a56147d615e196a81f57ddde552d4d92c396328f6c78c73eaabb15e"} Jul 16 21:41:49.407207 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.407017 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"042787c0-eece-41ee-b1ad-12bcd75922f4","Type":"ContainerStarted","Data":"e5ed32954cbdaae57691592bac02137d800e24936e38c4d6bdbc41932b6ff5b6"} Jul 16 21:41:49.437250 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.437195 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.422819538 podStartE2EDuration="2.43717529s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.355752ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:41:49.435805288 +0000 UTC m=+156.478029559" watchObservedRunningTime="2026-07-16 21:41:49.43717529 +0000 UTC m=+156.479399527" Jul 16 21:41:49.940308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.940266 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:41:49.940658 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.940625 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="prometheus" containerID="cri-o://809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" gracePeriod=600 Jul 16 21:41:49.940799 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.940669 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy" containerID="cri-o://2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" gracePeriod=600 Jul 16 21:41:49.940799 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.940690 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy-web" containerID="cri-o://9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" gracePeriod=600 Jul 16 21:41:49.940799 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.940692 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="thanos-sidecar" containerID="cri-o://40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" gracePeriod=600 Jul 16 21:41:49.940799 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.940709 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="config-reloader" containerID="cri-o://a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" gracePeriod=600 Jul 16 21:41:49.940799 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:49.940684 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" gracePeriod=600 Jul 16 21:41:50.222652 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.222625 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.277366 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.277319 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-thanos-prometheus-http-client-file\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.277574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.277390 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-metrics-client-ca\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.277574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.277413 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-tls-assets\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.277574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.277430 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-tls\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.277574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.277449 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-2\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.277574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.277467 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-1\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.277877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.277818 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:50.278046 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278025 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-config-out\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278164 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278123 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:50.278164 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278159 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-db\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278345 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278190 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-grpc-tls\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278345 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278224 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-trusted-ca-bundle\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278345 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278223 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:50.278345 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278263 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-config\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278345 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278293 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch27q\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-kube-api-access-ch27q\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278345 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278337 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-web-config\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278764 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278375 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-serving-certs-ca-bundle\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278764 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278417 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-kubelet-serving-ca-bundle\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278764 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278446 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278764 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278471 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-metrics-client-certs\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278764 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278512 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-0\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278764 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278574 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-kube-rbac-proxy\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.278764 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278600 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2630e476-43b2-4db8-be71-62971d5e164c\" (UID: \"2630e476-43b2-4db8-be71-62971d5e164c\") " Jul 16 21:41:50.279125 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278910 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-metrics-client-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.279125 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278929 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.279125 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.278944 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.279899 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.279622 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:50.279899 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.279778 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:41:50.280165 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.280138 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.282474 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.282442 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.285002 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.282858 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:50.285002 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.284837 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:50.286263 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.285710 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-config-out" (OuterVolumeSpecName: "config-out") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:41:50.286263 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.285796 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:41:50.286263 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.285863 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-kube-api-access-ch27q" (OuterVolumeSpecName: "kube-api-access-ch27q") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "kube-api-access-ch27q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:41:50.286263 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.285878 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.286263 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.286189 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.286909 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.286793 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:50.286909 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.286871 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.287584 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.287437 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-config" (OuterVolumeSpecName: "config") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.287827 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.287780 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.288794 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.288764 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.300891 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.300855 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-web-config" (OuterVolumeSpecName: "web-config") pod "2630e476-43b2-4db8-be71-62971d5e164c" (UID: "2630e476-43b2-4db8-be71-62971d5e164c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.380149 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380109 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380409 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380390 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380559 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380510 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380559 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380555 2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-metrics-client-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380559 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380565 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380575 2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-kube-rbac-proxy\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380584 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380596 2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380610 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-tls-assets\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380621 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380630 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-config-out\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380640 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-k8s-db\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380653 2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-secret-grpc-tls\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380665 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2630e476-43b2-4db8-be71-62971d5e164c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380675 2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380683 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch27q\" (UniqueName: \"kubernetes.io/projected/2630e476-43b2-4db8-be71-62971d5e164c-kube-api-access-ch27q\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.380747 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.380692 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2630e476-43b2-4db8-be71-62971d5e164c-web-config\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.415147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415102 2573 generic.go:356] "Generic (PLEG): container finished" podID="2630e476-43b2-4db8-be71-62971d5e164c" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" exitCode=0 Jul 16 21:41:50.415147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415130 2573 generic.go:356] "Generic (PLEG): container finished" podID="2630e476-43b2-4db8-be71-62971d5e164c" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" exitCode=0 Jul 16 21:41:50.415147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415136 2573 generic.go:356] "Generic (PLEG): container finished" podID="2630e476-43b2-4db8-be71-62971d5e164c" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" exitCode=0 Jul 16 21:41:50.415147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415142 2573 generic.go:356] "Generic (PLEG): container finished" podID="2630e476-43b2-4db8-be71-62971d5e164c" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" exitCode=0 Jul 16 21:41:50.415147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415147 2573 generic.go:356] "Generic (PLEG): container finished" podID="2630e476-43b2-4db8-be71-62971d5e164c" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" exitCode=0 Jul 16 21:41:50.415147 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415153 2573 generic.go:356] "Generic (PLEG): container finished" podID="2630e476-43b2-4db8-be71-62971d5e164c" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" exitCode=0 Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerDied","Data":"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf"} Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415206 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415287 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerDied","Data":"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8"} Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerDied","Data":"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814"} Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415346 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerDied","Data":"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd"} Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415352 2573 scope.go:117] "RemoveContainer" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415370 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerDied","Data":"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c"} Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415394 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerDied","Data":"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b"} Jul 16 21:41:50.415877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.415421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2630e476-43b2-4db8-be71-62971d5e164c","Type":"ContainerDied","Data":"f6b47631a3a8b53c93061d39aff3a23b521ee7db0ffb8eb712d2a21251d46f03"} Jul 16 21:41:50.424249 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.424227 2573 scope.go:117] "RemoveContainer" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" Jul 16 21:41:50.431932 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.431915 2573 scope.go:117] "RemoveContainer" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" Jul 16 21:41:50.440370 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.440335 2573 scope.go:117] "RemoveContainer" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" Jul 16 21:41:50.440836 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.440813 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:41:50.445954 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.445927 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:41:50.448981 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.448959 2573 scope.go:117] "RemoveContainer" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" Jul 16 21:41:50.456673 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.456653 2573 scope.go:117] "RemoveContainer" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" Jul 16 21:41:50.464726 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.464704 2573 scope.go:117] "RemoveContainer" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" Jul 16 21:41:50.472408 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.472388 2573 scope.go:117] "RemoveContainer" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" Jul 16 21:41:50.472731 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:50.472712 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": container with ID starting with aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf not found: ID does not exist" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" Jul 16 21:41:50.472807 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.472740 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf"} err="failed to get container status \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": rpc error: code = NotFound desc = could not find container \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": container with ID starting with aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf not found: ID does not exist" Jul 16 21:41:50.472807 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.472759 2573 scope.go:117] "RemoveContainer" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" Jul 16 21:41:50.473047 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:50.473025 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": container with ID starting with 2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8 not found: ID does not exist" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" Jul 16 21:41:50.473089 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.473054 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8"} err="failed to get container status \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": rpc error: code = NotFound desc = could not find container \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": container with ID starting with 2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8 not found: ID does not exist" Jul 16 21:41:50.473089 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.473071 2573 scope.go:117] "RemoveContainer" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" Jul 16 21:41:50.473320 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:50.473303 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": container with ID starting with 9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814 not found: ID does not exist" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" Jul 16 21:41:50.473374 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.473325 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814"} err="failed to get container status \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": rpc error: code = NotFound desc = could not find container \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": container with ID starting with 9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814 not found: ID does not exist" Jul 16 21:41:50.473374 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.473340 2573 scope.go:117] "RemoveContainer" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" Jul 16 21:41:50.473570 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:50.473556 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": container with ID starting with 40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd not found: ID does not exist" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" Jul 16 21:41:50.473626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.473575 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd"} err="failed to get container status \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": rpc error: code = NotFound desc = could not find container \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": container with ID starting with 40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd not found: ID does not exist" Jul 16 21:41:50.473626 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.473593 2573 scope.go:117] "RemoveContainer" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" Jul 16 21:41:50.473843 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:50.473823 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": container with ID starting with a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c not found: ID does not exist" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" Jul 16 21:41:50.473889 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.473847 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c"} err="failed to get container status \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": rpc error: code = NotFound desc = could not find container \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": container with ID starting with a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c not found: ID does not exist" Jul 16 21:41:50.473889 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.473862 2573 scope.go:117] "RemoveContainer" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" Jul 16 21:41:50.474081 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:50.474064 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": container with ID starting with 809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b not found: ID does not exist" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" Jul 16 21:41:50.474120 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.474083 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b"} err="failed to get container status \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": rpc error: code = NotFound desc = could not find container \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": container with ID starting with 809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b not found: ID does not exist" Jul 16 21:41:50.474120 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.474096 2573 scope.go:117] "RemoveContainer" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" Jul 16 21:41:50.474315 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:41:50.474295 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": container with ID starting with 15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2 not found: ID does not exist" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" Jul 16 21:41:50.474351 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.474319 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2"} err="failed to get container status \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": rpc error: code = NotFound desc = could not find container \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": container with ID starting with 15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2 not found: ID does not exist" Jul 16 21:41:50.474351 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.474338 2573 scope.go:117] "RemoveContainer" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" Jul 16 21:41:50.474881 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.474862 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf"} err="failed to get container status \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": rpc error: code = NotFound desc = could not find container \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": container with ID starting with aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf not found: ID does not exist" Jul 16 21:41:50.474928 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.474883 2573 scope.go:117] "RemoveContainer" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" Jul 16 21:41:50.475115 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.475097 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8"} err="failed to get container status \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": rpc error: code = NotFound desc = could not find container \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": container with ID starting with 2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8 not found: ID does not exist" Jul 16 21:41:50.475170 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.475115 2573 scope.go:117] "RemoveContainer" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" Jul 16 21:41:50.475307 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.475291 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814"} err="failed to get container status \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": rpc error: code = NotFound desc = could not find container \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": container with ID starting with 9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814 not found: ID does not exist" Jul 16 21:41:50.475355 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.475307 2573 scope.go:117] "RemoveContainer" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" Jul 16 21:41:50.475715 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.475680 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd"} err="failed to get container status \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": rpc error: code = NotFound desc = could not find container \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": container with ID starting with 40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd not found: ID does not exist" Jul 16 21:41:50.475715 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.475703 2573 scope.go:117] "RemoveContainer" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" Jul 16 21:41:50.476427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.475986 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c"} err="failed to get container status \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": rpc error: code = NotFound desc = could not find container \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": container with ID starting with a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c not found: ID does not exist" Jul 16 21:41:50.476427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.476013 2573 scope.go:117] "RemoveContainer" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" Jul 16 21:41:50.476427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.476314 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b"} err="failed to get container status \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": rpc error: code = NotFound desc = could not find container \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": container with ID starting with 809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b not found: ID does not exist" Jul 16 21:41:50.476427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.476336 2573 scope.go:117] "RemoveContainer" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" Jul 16 21:41:50.476662 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.476630 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2"} err="failed to get container status \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": rpc error: code = NotFound desc = could not find container \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": container with ID starting with 15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2 not found: ID does not exist" Jul 16 21:41:50.476662 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.476647 2573 scope.go:117] "RemoveContainer" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" Jul 16 21:41:50.476886 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.476868 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf"} err="failed to get container status \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": rpc error: code = NotFound desc = could not find container \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": container with ID starting with aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf not found: ID does not exist" Jul 16 21:41:50.476886 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.476886 2573 scope.go:117] "RemoveContainer" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" Jul 16 21:41:50.477116 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.477095 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8"} err="failed to get container status \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": rpc error: code = NotFound desc = could not find container \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": container with ID starting with 2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8 not found: ID does not exist" Jul 16 21:41:50.477116 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.477114 2573 scope.go:117] "RemoveContainer" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" Jul 16 21:41:50.477341 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.477324 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814"} err="failed to get container status \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": rpc error: code = NotFound desc = could not find container \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": container with ID starting with 9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814 not found: ID does not exist" Jul 16 21:41:50.477406 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.477343 2573 scope.go:117] "RemoveContainer" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" Jul 16 21:41:50.477661 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.477638 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd"} err="failed to get container status \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": rpc error: code = NotFound desc = could not find container \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": container with ID starting with 40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd not found: ID does not exist" Jul 16 21:41:50.477661 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.477661 2573 scope.go:117] "RemoveContainer" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" Jul 16 21:41:50.477920 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.477897 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c"} err="failed to get container status \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": rpc error: code = NotFound desc = could not find container \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": container with ID starting with a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c not found: ID does not exist" Jul 16 21:41:50.477972 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.477922 2573 scope.go:117] "RemoveContainer" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" Jul 16 21:41:50.478164 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478146 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b"} err="failed to get container status \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": rpc error: code = NotFound desc = could not find container \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": container with ID starting with 809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b not found: ID does not exist" Jul 16 21:41:50.478263 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478165 2573 scope.go:117] "RemoveContainer" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" Jul 16 21:41:50.478263 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478203 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:41:50.478411 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478393 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2"} err="failed to get container status \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": rpc error: code = NotFound desc = could not find container \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": container with ID starting with 15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2 not found: ID does not exist" Jul 16 21:41:50.478447 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478412 2573 scope.go:117] "RemoveContainer" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" Jul 16 21:41:50.478551 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478535 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy" Jul 16 21:41:50.478611 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478554 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy" Jul 16 21:41:50.478611 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478570 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy-web" Jul 16 21:41:50.478611 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478576 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy-web" Jul 16 21:41:50.478611 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478592 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="init-config-reloader" Jul 16 21:41:50.478611 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478601 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="init-config-reloader" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478615 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="config-reloader" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478624 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="config-reloader" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478632 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="thanos-sidecar" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478642 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="thanos-sidecar" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478651 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy-thanos" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478656 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy-thanos" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478664 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="prometheus" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478669 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="prometheus" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478626 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf"} err="failed to get container status \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": rpc error: code = NotFound desc = could not find container \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": container with ID starting with aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf not found: ID does not exist" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478714 2573 scope.go:117] "RemoveContainer" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478734 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="thanos-sidecar" Jul 16 21:41:50.478753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478748 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy-thanos" Jul 16 21:41:50.479217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478760 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy" Jul 16 21:41:50.479217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478769 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="kube-rbac-proxy-web" Jul 16 21:41:50.479217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478779 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="config-reloader" Jul 16 21:41:50.479217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478785 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2630e476-43b2-4db8-be71-62971d5e164c" containerName="prometheus" Jul 16 21:41:50.479217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478957 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8"} err="failed to get container status \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": rpc error: code = NotFound desc = could not find container \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": container with ID starting with 2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8 not found: ID does not exist" Jul 16 21:41:50.479217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.478977 2573 scope.go:117] "RemoveContainer" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" Jul 16 21:41:50.479739 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.479708 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814"} err="failed to get container status \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": rpc error: code = NotFound desc = could not find container \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": container with ID starting with 9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814 not found: ID does not exist" Jul 16 21:41:50.479739 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.479733 2573 scope.go:117] "RemoveContainer" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" Jul 16 21:41:50.480048 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.480024 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd"} err="failed to get container status \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": rpc error: code = NotFound desc = could not find container \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": container with ID starting with 40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd not found: ID does not exist" Jul 16 21:41:50.480135 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.480048 2573 scope.go:117] "RemoveContainer" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" Jul 16 21:41:50.480358 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.480318 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c"} err="failed to get container status \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": rpc error: code = NotFound desc = could not find container \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": container with ID starting with a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c not found: ID does not exist" Jul 16 21:41:50.480358 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.480353 2573 scope.go:117] "RemoveContainer" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" Jul 16 21:41:50.480630 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.480613 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b"} err="failed to get container status \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": rpc error: code = NotFound desc = could not find container \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": container with ID starting with 809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b not found: ID does not exist" Jul 16 21:41:50.480630 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.480630 2573 scope.go:117] "RemoveContainer" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" Jul 16 21:41:50.480893 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.480872 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2"} err="failed to get container status \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": rpc error: code = NotFound desc = could not find container \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": container with ID starting with 15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2 not found: ID does not exist" Jul 16 21:41:50.480940 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.480895 2573 scope.go:117] "RemoveContainer" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" Jul 16 21:41:50.481513 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.481163 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf"} err="failed to get container status \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": rpc error: code = NotFound desc = could not find container \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": container with ID starting with aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf not found: ID does not exist" Jul 16 21:41:50.481513 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.481190 2573 scope.go:117] "RemoveContainer" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" Jul 16 21:41:50.481513 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.481476 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8"} err="failed to get container status \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": rpc error: code = NotFound desc = could not find container \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": container with ID starting with 2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8 not found: ID does not exist" Jul 16 21:41:50.481513 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.481499 2573 scope.go:117] "RemoveContainer" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" Jul 16 21:41:50.481812 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.481768 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814"} err="failed to get container status \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": rpc error: code = NotFound desc = could not find container \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": container with ID starting with 9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814 not found: ID does not exist" Jul 16 21:41:50.481812 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.481784 2573 scope.go:117] "RemoveContainer" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" Jul 16 21:41:50.482022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482003 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd"} err="failed to get container status \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": rpc error: code = NotFound desc = could not find container \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": container with ID starting with 40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd not found: ID does not exist" Jul 16 21:41:50.482089 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482023 2573 scope.go:117] "RemoveContainer" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" Jul 16 21:41:50.482260 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482242 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c"} err="failed to get container status \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": rpc error: code = NotFound desc = could not find container \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": container with ID starting with a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c not found: ID does not exist" Jul 16 21:41:50.482302 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482262 2573 scope.go:117] "RemoveContainer" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" Jul 16 21:41:50.482471 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482455 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b"} err="failed to get container status \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": rpc error: code = NotFound desc = could not find container \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": container with ID starting with 809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b not found: ID does not exist" Jul 16 21:41:50.482554 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482473 2573 scope.go:117] "RemoveContainer" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" Jul 16 21:41:50.482708 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482691 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2"} err="failed to get container status \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": rpc error: code = NotFound desc = could not find container \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": container with ID starting with 15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2 not found: ID does not exist" Jul 16 21:41:50.482768 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482709 2573 scope.go:117] "RemoveContainer" containerID="aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf" Jul 16 21:41:50.482964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.482964 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482955 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf"} err="failed to get container status \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": rpc error: code = NotFound desc = could not find container \"aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf\": container with ID starting with aba314a6000f3e09700623b6ba7e3ba44ea7c8a756eabfbeedd8cdefbf8cd9bf not found: ID does not exist" Jul 16 21:41:50.483063 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.482972 2573 scope.go:117] "RemoveContainer" containerID="2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8" Jul 16 21:41:50.483348 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.483177 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8"} err="failed to get container status \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": rpc error: code = NotFound desc = could not find container \"2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8\": container with ID starting with 2dd085eeea4e41494de112a77ba9fbc886a840f07b5ee303fa801995640ddde8 not found: ID does not exist" Jul 16 21:41:50.483348 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.483202 2573 scope.go:117] "RemoveContainer" containerID="9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814" Jul 16 21:41:50.483657 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.483565 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814"} err="failed to get container status \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": rpc error: code = NotFound desc = could not find container \"9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814\": container with ID starting with 9b688b9af4ff3f815b14196518a45be45fa42d85946d8ba21c84a26673cfd814 not found: ID does not exist" Jul 16 21:41:50.483746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.483662 2573 scope.go:117] "RemoveContainer" containerID="40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd" Jul 16 21:41:50.483966 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.483937 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd"} err="failed to get container status \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": rpc error: code = NotFound desc = could not find container \"40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd\": container with ID starting with 40a0e019ff1d2c75e7cf26730a5784e13bea702d3b737e1449725d4fef77bbcd not found: ID does not exist" Jul 16 21:41:50.484041 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.483970 2573 scope.go:117] "RemoveContainer" containerID="a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c" Jul 16 21:41:50.484240 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.484220 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c"} err="failed to get container status \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": rpc error: code = NotFound desc = could not find container \"a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c\": container with ID starting with a32ab707143111b206f50f6ac82f23f0de51c42211af0f0b40bc879bb1ceab6c not found: ID does not exist" Jul 16 21:41:50.484240 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.484240 2573 scope.go:117] "RemoveContainer" containerID="809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b" Jul 16 21:41:50.484501 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.484474 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b"} err="failed to get container status \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": rpc error: code = NotFound desc = could not find container \"809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b\": container with ID starting with 809e480ba73ea520f49bed37eeca0974cfac1b627bf109c33e065d5df1ae752b not found: ID does not exist" Jul 16 21:41:50.484638 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.484503 2573 scope.go:117] "RemoveContainer" containerID="15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2" Jul 16 21:41:50.484842 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.484817 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2"} err="failed to get container status \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": rpc error: code = NotFound desc = could not find container \"15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2\": container with ID starting with 15a2c5b85cda86ba2e2ecc534f59b9164842375431fac28dcfcd454a39037fe2 not found: ID does not exist" Jul 16 21:41:50.485830 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.485813 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 21:41:50.486021 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486003 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 21:41:50.486158 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486010 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 21:41:50.486158 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486116 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 21:41:50.486285 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486158 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 21:41:50.486325 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486298 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 21:41:50.486377 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486363 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 21:41:50.486409 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486297 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q7x7k\"" Jul 16 21:41:50.486409 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486397 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 21:41:50.486470 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486446 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 21:41:50.486911 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486896 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 21:41:50.487008 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.486907 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 21:41:50.487078 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.487058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1pkqkt4vsgplp\"" Jul 16 21:41:50.487137 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.487107 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 21:41:50.488743 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.488724 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 21:41:50.492638 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.492617 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 21:41:50.502853 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.502824 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:41:50.581962 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.581925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.581962 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.581968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582174 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582174 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582174 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582174 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-config\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582174 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-web-config\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdszr\" (UniqueName: \"kubernetes.io/projected/87572c86-25d0-48aa-89a2-8d4fe36e616b-kube-api-access-vdszr\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582389 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582728 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582728 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582728 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87572c86-25d0-48aa-89a2-8d4fe36e616b-config-out\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582728 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582504 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87572c86-25d0-48aa-89a2-8d4fe36e616b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.582728 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.582559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683322 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683322 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-config\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-web-config\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683939 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdszr\" (UniqueName: \"kubernetes.io/projected/87572c86-25d0-48aa-89a2-8d4fe36e616b-kube-api-access-vdszr\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683939 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683939 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.683939 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684140 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684140 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.683985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87572c86-25d0-48aa-89a2-8d4fe36e616b-config-out\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684140 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.684019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87572c86-25d0-48aa-89a2-8d4fe36e616b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684140 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.684042 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684140 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.684043 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684140 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.684091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684140 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.684120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684465 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.684152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684465 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.684249 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.684577 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.684487 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.687704 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.685254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.687704 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.686738 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.687704 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.687143 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.687704 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.687361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-web-config\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.687704 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.687507 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.687704 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.687640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.687704 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.687661 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.688140 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.687764 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-config\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.688402 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.688370 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.688545 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.688438 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.688659 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.688640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.688703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.688645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87572c86-25d0-48aa-89a2-8d4fe36e616b-config-out\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.688703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.688641 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87572c86-25d0-48aa-89a2-8d4fe36e616b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.689095 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.689076 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.689385 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.689371 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/87572c86-25d0-48aa-89a2-8d4fe36e616b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.689682 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.689667 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87572c86-25d0-48aa-89a2-8d4fe36e616b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.692712 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.692689 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdszr\" (UniqueName: \"kubernetes.io/projected/87572c86-25d0-48aa-89a2-8d4fe36e616b-kube-api-access-vdszr\") pod \"prometheus-k8s-0\" (UID: \"87572c86-25d0-48aa-89a2-8d4fe36e616b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.793784 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.793685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:41:50.955619 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:50.955274 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:41:50.958460 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:41:50.958426 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87572c86_25d0_48aa_89a2_8d4fe36e616b.slice/crio-2e391dc363229685e4fa101472ba4b65c5848fdc821b167f37c05da01b47db19 WatchSource:0}: Error finding container 2e391dc363229685e4fa101472ba4b65c5848fdc821b167f37c05da01b47db19: Status 404 returned error can't find the container with id 2e391dc363229685e4fa101472ba4b65c5848fdc821b167f37c05da01b47db19 Jul 16 21:41:51.422632 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:51.422541 2573 generic.go:356] "Generic (PLEG): container finished" podID="87572c86-25d0-48aa-89a2-8d4fe36e616b" containerID="f817ce5bc19595d1c545a5f66b78841c76a9ae57e7035dbfe0565e19b8d4235e" exitCode=0 Jul 16 21:41:51.422632 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:51.422587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87572c86-25d0-48aa-89a2-8d4fe36e616b","Type":"ContainerDied","Data":"f817ce5bc19595d1c545a5f66b78841c76a9ae57e7035dbfe0565e19b8d4235e"} Jul 16 21:41:51.423019 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:51.422662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87572c86-25d0-48aa-89a2-8d4fe36e616b","Type":"ContainerStarted","Data":"2e391dc363229685e4fa101472ba4b65c5848fdc821b167f37c05da01b47db19"} Jul 16 21:41:51.543134 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:51.543104 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2630e476-43b2-4db8-be71-62971d5e164c" path="/var/lib/kubelet/pods/2630e476-43b2-4db8-be71-62971d5e164c/volumes" Jul 16 21:41:52.432688 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:52.432653 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87572c86-25d0-48aa-89a2-8d4fe36e616b","Type":"ContainerStarted","Data":"544a3c64479fba07de0be01c81817d7d2e1e369920bda81ce7ed45b8a2878bbf"} Jul 16 21:41:52.433078 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:52.432719 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87572c86-25d0-48aa-89a2-8d4fe36e616b","Type":"ContainerStarted","Data":"805cb5b665be2901758aababca477cadd7aa38fbae53385e595872fed24d5057"} Jul 16 21:41:52.433078 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:52.432747 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87572c86-25d0-48aa-89a2-8d4fe36e616b","Type":"ContainerStarted","Data":"b3460ff096eff3ebad4abc6637d10e34e0278ecb6e50f96209dcbb221b415aff"} Jul 16 21:41:52.433078 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:52.432772 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87572c86-25d0-48aa-89a2-8d4fe36e616b","Type":"ContainerStarted","Data":"3d1b4a8ca6cca131ee258d9152c1cc669c501c4cb178f83808a753720a5a47fc"} Jul 16 21:41:52.433078 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:52.432796 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87572c86-25d0-48aa-89a2-8d4fe36e616b","Type":"ContainerStarted","Data":"44be87057d2328cf5bc1a7215e953da5e45453539ed4127787cbf08bdee3bada"} Jul 16 21:41:52.433078 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:52.432820 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87572c86-25d0-48aa-89a2-8d4fe36e616b","Type":"ContainerStarted","Data":"75dfaf20c26cc516c0036cea982bbfee5aaf5090fd1074d25aaea6a3f6dc4bd0"} Jul 16 21:41:52.463179 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:52.463119 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.443891656 podStartE2EDuration="2.463101671s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.210015ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:41:52.460381962 +0000 UTC m=+159.502606202" watchObservedRunningTime="2026-07-16 21:41:52.463101671 +0000 UTC m=+159.505325909" Jul 16 21:41:53.374786 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:53.374751 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:53.431933 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:53.431840 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:41:55.794030 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:41:55.793991 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:42:04.131248 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:04.131216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:04.189868 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:04.189827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:04.746347 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:04.746264 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:04.847631 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:04.847587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:04.947769 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:04.947731 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:14.806373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:14.806339 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:14.863552 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:14.863504 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:15.012446 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:15.012415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:15.068080 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:15.067986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:42:50.794316 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:50.794273 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:42:50.810463 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:50.810433 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:42:51.691879 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:51.691851 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:42:56.265020 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:42:56.264988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:01.618729 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:01.618684 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:01.708618 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:01.708576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:28.407796 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:28.407756 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:29.211923 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:29.211880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:32.023540 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:32.023470 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:32.932871 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:32.932839 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:34.378267 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:34.378233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:34.493977 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:34.493939 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:36.779838 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:36.779801 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:36.978456 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:36.978418 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:37.133684 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.133640 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mv8sm"] Jul 16 21:43:37.138586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.138556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:37.141121 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.141095 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 21:43:37.141328 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.141285 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 21:43:37.142255 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.142233 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rngdd\"" Jul 16 21:43:37.144058 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.144033 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mv8sm"] Jul 16 21:43:37.238755 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.238716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mv8sm\" (UID: \"bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:37.238755 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.238758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznv6\" (UniqueName: \"kubernetes.io/projected/bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d-kube-api-access-hznv6\") pod \"cert-manager-webhook-6486ffd997-mv8sm\" (UID: \"bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:37.339317 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.339270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mv8sm\" (UID: \"bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:37.339494 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.339322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznv6\" (UniqueName: \"kubernetes.io/projected/bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d-kube-api-access-hznv6\") pod \"cert-manager-webhook-6486ffd997-mv8sm\" (UID: \"bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:37.347921 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.347881 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mv8sm\" (UID: \"bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:37.348067 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.348051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznv6\" (UniqueName: \"kubernetes.io/projected/bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d-kube-api-access-hznv6\") pod \"cert-manager-webhook-6486ffd997-mv8sm\" (UID: \"bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:37.459034 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.458938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:37.591339 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.591302 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mv8sm"] Jul 16 21:43:37.594434 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:43:37.594394 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbf92f1d_41e6_4a82_b81c_5b55d37e6a8d.slice/crio-39db509f832cad681758f09cedede5da43a6b39f5a1c811175a797de35b13f15 WatchSource:0}: Error finding container 39db509f832cad681758f09cedede5da43a6b39f5a1c811175a797de35b13f15: Status 404 returned error can't find the container with id 39db509f832cad681758f09cedede5da43a6b39f5a1c811175a797de35b13f15 Jul 16 21:43:37.867926 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:37.867896 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" event={"ID":"bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d","Type":"ContainerStarted","Data":"39db509f832cad681758f09cedede5da43a6b39f5a1c811175a797de35b13f15"} Jul 16 21:43:38.609113 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.609042 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:38.665857 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.665805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:38.736102 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.735823 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd"] Jul 16 21:43:38.739752 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.739723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" Jul 16 21:43:38.742763 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.742546 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2flnc\"" Jul 16 21:43:38.753207 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.753153 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd"] Jul 16 21:43:38.851593 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.851549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmjn\" (UniqueName: \"kubernetes.io/projected/0dffd9be-7a40-4b9f-8428-066b08f60ea9-kube-api-access-ndmjn\") pod \"cert-manager-cainjector-559cdd8c87-t2qdd\" (UID: \"0dffd9be-7a40-4b9f-8428-066b08f60ea9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" Jul 16 21:43:38.851798 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.851623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dffd9be-7a40-4b9f-8428-066b08f60ea9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-t2qdd\" (UID: \"0dffd9be-7a40-4b9f-8428-066b08f60ea9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" Jul 16 21:43:38.952614 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.952514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dffd9be-7a40-4b9f-8428-066b08f60ea9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-t2qdd\" (UID: \"0dffd9be-7a40-4b9f-8428-066b08f60ea9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" Jul 16 21:43:38.953071 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.952680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmjn\" (UniqueName: \"kubernetes.io/projected/0dffd9be-7a40-4b9f-8428-066b08f60ea9-kube-api-access-ndmjn\") pod \"cert-manager-cainjector-559cdd8c87-t2qdd\" (UID: \"0dffd9be-7a40-4b9f-8428-066b08f60ea9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" Jul 16 21:43:38.964023 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.963977 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dffd9be-7a40-4b9f-8428-066b08f60ea9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-t2qdd\" (UID: \"0dffd9be-7a40-4b9f-8428-066b08f60ea9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" Jul 16 21:43:38.964356 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:38.964295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmjn\" (UniqueName: \"kubernetes.io/projected/0dffd9be-7a40-4b9f-8428-066b08f60ea9-kube-api-access-ndmjn\") pod \"cert-manager-cainjector-559cdd8c87-t2qdd\" (UID: \"0dffd9be-7a40-4b9f-8428-066b08f60ea9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" Jul 16 21:43:39.053788 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:39.053740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" Jul 16 21:43:40.745563 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:40.745503 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd"] Jul 16 21:43:40.748560 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:43:40.748533 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dffd9be_7a40_4b9f_8428_066b08f60ea9.slice/crio-15f87bed88cc16ad655d59a2184c49462dae1fa3ae4c06e9b1632231d3a8a372 WatchSource:0}: Error finding container 15f87bed88cc16ad655d59a2184c49462dae1fa3ae4c06e9b1632231d3a8a372: Status 404 returned error can't find the container with id 15f87bed88cc16ad655d59a2184c49462dae1fa3ae4c06e9b1632231d3a8a372 Jul 16 21:43:40.884544 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:40.884417 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" event={"ID":"bbf92f1d-41e6-4a82-b81c-5b55d37e6a8d","Type":"ContainerStarted","Data":"6de3c90fa96db4231882f69b6163d8b7c008dd88816aa4c3a06213c57dc7799d"} Jul 16 21:43:40.884817 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:40.884756 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:40.886618 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:40.886592 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" event={"ID":"0dffd9be-7a40-4b9f-8428-066b08f60ea9","Type":"ContainerStarted","Data":"9e45ab4f734d491ccffaac3aa8943e3619fffd0313b1436a8876e565a70976a6"} Jul 16 21:43:40.886748 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:40.886624 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" event={"ID":"0dffd9be-7a40-4b9f-8428-066b08f60ea9","Type":"ContainerStarted","Data":"15f87bed88cc16ad655d59a2184c49462dae1fa3ae4c06e9b1632231d3a8a372"} Jul 16 21:43:40.904191 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:40.904128 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" podStartSLOduration=0.820165787 podStartE2EDuration="3.904102631s" totalImagesPullingTime="3.083936844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:43:40.901790455 +0000 UTC m=+267.944014694" watchObservedRunningTime="2026-07-16 21:43:40.904102631 +0000 UTC m=+267.946326870" Jul 16 21:43:40.917970 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:40.917906 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-t2qdd" podStartSLOduration=2.9178892210000003 podStartE2EDuration="2.917889221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:43:40.916899077 +0000 UTC m=+267.959123317" watchObservedRunningTime="2026-07-16 21:43:40.917889221 +0000 UTC m=+267.960113459" Jul 16 21:43:44.374044 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:44.374005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:46.623507 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:46.623475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:46.722158 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:46.722125 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:46.893305 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:46.893209 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mv8sm" Jul 16 21:43:52.498999 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:52.498971 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:52.602898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:52.602863 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:43:54.498420 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:43:54.498371 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:04.775403 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:04.775352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:06.906399 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:06.906361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:07.823165 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:07.823128 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:07.945831 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:07.945797 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:08.642574 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:08.642547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:08.971238 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:08.971139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:09.071179 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.071114 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:09.135106 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.135062 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm"] Jul 16 21:44:09.137622 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.137603 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.145370 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.145343 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 21:44:09.145606 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.145588 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 21:44:09.145807 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.145791 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 21:44:09.146187 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.146173 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-htzmn\"" Jul 16 21:44:09.149320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.149305 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 21:44:09.168932 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.168897 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm"] Jul 16 21:44:09.276761 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.276667 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:09.325923 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.325873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67a5213d-5613-4946-a352-8ae8f2443346-manifests\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.326088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.325959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67a5213d-5613-4946-a352-8ae8f2443346-apiservice-cert\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.326088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.325992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pf2t\" (UniqueName: \"kubernetes.io/projected/67a5213d-5613-4946-a352-8ae8f2443346-kube-api-access-8pf2t\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.326088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.326060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a5213d-5613-4946-a352-8ae8f2443346-webhook-cert\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.326204 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.326100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a5213d-5613-4946-a352-8ae8f2443346-tmp\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.376808 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.376768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:09.426972 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.426931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a5213d-5613-4946-a352-8ae8f2443346-webhook-cert\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.427177 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.426983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a5213d-5613-4946-a352-8ae8f2443346-tmp\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.427177 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.427013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67a5213d-5613-4946-a352-8ae8f2443346-manifests\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.427177 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.427035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67a5213d-5613-4946-a352-8ae8f2443346-apiservice-cert\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.427177 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.427055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pf2t\" (UniqueName: \"kubernetes.io/projected/67a5213d-5613-4946-a352-8ae8f2443346-kube-api-access-8pf2t\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.427412 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.427384 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67a5213d-5613-4946-a352-8ae8f2443346-manifests\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.429552 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.429494 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a5213d-5613-4946-a352-8ae8f2443346-tmp\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.429685 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.429641 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a5213d-5613-4946-a352-8ae8f2443346-webhook-cert\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.429735 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.429709 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67a5213d-5613-4946-a352-8ae8f2443346-apiservice-cert\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.438845 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.438814 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pf2t\" (UniqueName: \"kubernetes.io/projected/67a5213d-5613-4946-a352-8ae8f2443346-kube-api-access-8pf2t\") pod \"opendatahub-operator-controller-manager-db5748bcb-6vwzm\" (UID: \"67a5213d-5613-4946-a352-8ae8f2443346\") " pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.448727 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.448692 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:09.477376 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.477346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:09.600792 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:09.600765 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm"] Jul 16 21:44:09.603448 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:44:09.603422 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a5213d_5613_4946_a352_8ae8f2443346.slice/crio-112cbad9d801c5d6a3a0c3636fea61fb9a15d6495d5a203ee3c04567b6144aad WatchSource:0}: Error finding container 112cbad9d801c5d6a3a0c3636fea61fb9a15d6495d5a203ee3c04567b6144aad: Status 404 returned error can't find the container with id 112cbad9d801c5d6a3a0c3636fea61fb9a15d6495d5a203ee3c04567b6144aad Jul 16 21:44:10.015812 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:10.015779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" event={"ID":"67a5213d-5613-4946-a352-8ae8f2443346","Type":"ContainerStarted","Data":"112cbad9d801c5d6a3a0c3636fea61fb9a15d6495d5a203ee3c04567b6144aad"} Jul 16 21:44:10.157183 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:10.157156 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:10.259762 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:10.259724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:13.031561 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:13.031497 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" event={"ID":"67a5213d-5613-4946-a352-8ae8f2443346","Type":"ContainerStarted","Data":"8b35476e9989f0fc10099f4daf3352018d77ac88c6d7a1640d00a2d4651eaf4b"} Jul 16 21:44:13.332040 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:13.332012 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 21:44:14.037039 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:14.036948 2573 generic.go:356] "Generic (PLEG): container finished" podID="67a5213d-5613-4946-a352-8ae8f2443346" containerID="8b35476e9989f0fc10099f4daf3352018d77ac88c6d7a1640d00a2d4651eaf4b" exitCode=0 Jul 16 21:44:14.037039 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:14.037008 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" event={"ID":"67a5213d-5613-4946-a352-8ae8f2443346","Type":"ContainerDied","Data":"8b35476e9989f0fc10099f4daf3352018d77ac88c6d7a1640d00a2d4651eaf4b"} Jul 16 21:44:16.048375 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:16.048342 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" event={"ID":"67a5213d-5613-4946-a352-8ae8f2443346","Type":"ContainerStarted","Data":"283b621084e2b98333f6edd42b8aba7d9cf58878774cb58de6689d24fe1aadb2"} Jul 16 21:44:16.048820 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:16.048504 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:16.076942 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:16.076890 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" podStartSLOduration=1.9825718490000002 podStartE2EDuration="7.076873343s" totalImagesPullingTime="3.936704563s" totalInitContainerRuntime="1.157596931s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:44:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:44:16.075957006 +0000 UTC m=+303.118181259" watchObservedRunningTime="2026-07-16 21:44:16.076873343 +0000 UTC m=+303.119097580" Jul 16 21:44:20.839234 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:20.839193 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:20.939452 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:20.939412 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:22.622372 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:22.622339 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:22.714246 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:22.714209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:26.444541 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.444488 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh"] Jul 16 21:44:26.452839 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.452813 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.455725 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.455702 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 21:44:26.456701 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.456680 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 21:44:26.456829 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.456684 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:44:26.456898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.456877 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 21:44:26.456970 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.456958 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v2t65\"" Jul 16 21:44:26.457014 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.456976 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 21:44:26.460457 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.460436 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh"] Jul 16 21:44:26.579475 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.579437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac89078a-70a8-4849-a0b1-5a18840bf108-manager-config\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.579685 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.579496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lx76\" (UniqueName: \"kubernetes.io/projected/ac89078a-70a8-4849-a0b1-5a18840bf108-kube-api-access-9lx76\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.579685 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.579622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac89078a-70a8-4849-a0b1-5a18840bf108-cert\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.579685 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.579673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac89078a-70a8-4849-a0b1-5a18840bf108-metrics-cert\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.680178 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.680146 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac89078a-70a8-4849-a0b1-5a18840bf108-metrics-cert\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.680354 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.680187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac89078a-70a8-4849-a0b1-5a18840bf108-manager-config\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.680354 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.680323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lx76\" (UniqueName: \"kubernetes.io/projected/ac89078a-70a8-4849-a0b1-5a18840bf108-kube-api-access-9lx76\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.680466 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.680411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac89078a-70a8-4849-a0b1-5a18840bf108-cert\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.680889 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.680865 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac89078a-70a8-4849-a0b1-5a18840bf108-manager-config\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.682753 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.682721 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac89078a-70a8-4849-a0b1-5a18840bf108-metrics-cert\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.682866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.682802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac89078a-70a8-4849-a0b1-5a18840bf108-cert\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.688744 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.688717 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lx76\" (UniqueName: \"kubernetes.io/projected/ac89078a-70a8-4849-a0b1-5a18840bf108-kube-api-access-9lx76\") pod \"lws-controller-manager-6844f9fcbb-2drxh\" (UID: \"ac89078a-70a8-4849-a0b1-5a18840bf108\") " pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.763216 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.763125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:26.898042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.898003 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh"] Jul 16 21:44:26.900726 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:44:26.900697 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac89078a_70a8_4849_a0b1_5a18840bf108.slice/crio-3311c7c031ea07f17d6be4f10e47f24d280cc0445a30daf3fb52d41dae539b0b WatchSource:0}: Error finding container 3311c7c031ea07f17d6be4f10e47f24d280cc0445a30daf3fb52d41dae539b0b: Status 404 returned error can't find the container with id 3311c7c031ea07f17d6be4f10e47f24d280cc0445a30daf3fb52d41dae539b0b Jul 16 21:44:26.903297 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:26.903281 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:44:27.055668 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:27.055641 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-db5748bcb-6vwzm" Jul 16 21:44:27.103159 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:27.103125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" event={"ID":"ac89078a-70a8-4849-a0b1-5a18840bf108","Type":"ContainerStarted","Data":"3311c7c031ea07f17d6be4f10e47f24d280cc0445a30daf3fb52d41dae539b0b"} Jul 16 21:44:28.761271 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:28.761234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:29.013289 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.013182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:29.320952 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.320918 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:29.383740 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.383689 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5"] Jul 16 21:44:29.394399 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.394359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:29.398506 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.398478 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 21:44:29.398668 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.398553 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-d67sv\"" Jul 16 21:44:29.398668 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.398597 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 21:44:29.398797 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.398689 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 21:44:29.399087 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.399071 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 21:44:29.408537 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.408499 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5"] Jul 16 21:44:29.508888 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.508854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tls-certs\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:29.509075 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.508910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tmp\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:29.509075 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.508937 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47dw\" (UniqueName: \"kubernetes.io/projected/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-kube-api-access-l47dw\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:29.610335 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.610238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tls-certs\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:29.610335 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.610314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tmp\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:29.610335 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.610337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47dw\" (UniqueName: \"kubernetes.io/projected/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-kube-api-access-l47dw\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:29.610651 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:44:29.610367 2573 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 16 21:44:29.610651 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:44:29.610445 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tls-certs podName:681b65a0-b05d-43c0-8ebf-8e1483b2bd26 nodeName:}" failed. No retries permitted until 2026-07-16 21:44:30.110420716 +0000 UTC m=+317.152644934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tls-certs") pod "kube-auth-proxy-67b5c5c797-ztsd5" (UID: "681b65a0-b05d-43c0-8ebf-8e1483b2bd26") : secret "kube-auth-proxy-tls" not found Jul 16 21:44:29.612771 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.612743 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tmp\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:29.628813 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:29.628786 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47dw\" (UniqueName: \"kubernetes.io/projected/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-kube-api-access-l47dw\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:30.067763 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.067731 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:30.115980 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.115947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tls-certs\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:30.118745 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.118715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681b65a0-b05d-43c0-8ebf-8e1483b2bd26-tls-certs\") pod \"kube-auth-proxy-67b5c5c797-ztsd5\" (UID: \"681b65a0-b05d-43c0-8ebf-8e1483b2bd26\") " pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:30.121908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.121378 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" event={"ID":"ac89078a-70a8-4849-a0b1-5a18840bf108","Type":"ContainerStarted","Data":"2d8db1ad1ff2a2a5beef73a8a9b1b84d9e0525badffcd8bec9eab56ad026b660"} Jul 16 21:44:30.121908 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.121589 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:30.307419 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.307382 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" Jul 16 21:44:30.316988 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.316961 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:30.512377 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.512316 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" podStartSLOduration=1.457381596 podStartE2EDuration="4.512295789s" totalImagesPullingTime="3.054914193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:44:30.211686863 +0000 UTC m=+317.253911101" watchObservedRunningTime="2026-07-16 21:44:30.512295789 +0000 UTC m=+317.554520004" Jul 16 21:44:30.513483 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.513422 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5"] Jul 16 21:44:30.516064 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:44:30.516041 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681b65a0_b05d_43c0_8ebf_8e1483b2bd26.slice/crio-07bc480d391db2f93ee366aba2e05d5cecdd486ab7fdb4a4d1d68b45a392a8e9 WatchSource:0}: Error finding container 07bc480d391db2f93ee366aba2e05d5cecdd486ab7fdb4a4d1d68b45a392a8e9: Status 404 returned error can't find the container with id 07bc480d391db2f93ee366aba2e05d5cecdd486ab7fdb4a4d1d68b45a392a8e9 Jul 16 21:44:30.545233 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.545204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:30.797958 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:30.797930 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:31.128101 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:31.128012 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" event={"ID":"681b65a0-b05d-43c0-8ebf-8e1483b2bd26","Type":"ContainerStarted","Data":"07bc480d391db2f93ee366aba2e05d5cecdd486ab7fdb4a4d1d68b45a392a8e9"} Jul 16 21:44:31.451889 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:31.451796 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:31.960915 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:31.960871 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:32.101108 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:32.101064 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:32.211645 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:32.211549 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:33.783067 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:33.783026 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:34.145777 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:34.145744 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" event={"ID":"681b65a0-b05d-43c0-8ebf-8e1483b2bd26","Type":"ContainerStarted","Data":"73bb566b76d85449e292f9d80c4eaac027db7f2f57e00eb0b63a25754cfadf29"} Jul 16 21:44:34.164216 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:34.164159 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67b5c5c797-ztsd5" podStartSLOduration=2.586471286 podStartE2EDuration="5.164142617s" totalImagesPullingTime="2.577671331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:44:34.162976105 +0000 UTC m=+321.205200371" watchObservedRunningTime="2026-07-16 21:44:34.164142617 +0000 UTC m=+321.206366854" Jul 16 21:44:38.267884 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:38.267842 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:38.462839 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:38.462797 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:39.640022 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:39.639980 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:39.963710 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:39.963623 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:40.162632 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:40.162584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:41.103813 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:41.103774 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:41.130392 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:41.130360 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6844f9fcbb-2drxh" Jul 16 21:44:41.260871 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:41.260839 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:41.602222 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:41.602181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:41.768715 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:41.768683 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:44:42.891854 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:44:42.891812 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:45:30.270944 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:45:30.270851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:45:30.361937 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:45:30.361900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:45:30.493776 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:45:30.493734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:45:45.747630 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:45:45.747594 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:45:58.715132 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:45:58.715098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:45:59.305440 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:45:59.305399 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:00.355355 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:00.355324 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:03.946477 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:03.946429 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:04.135820 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.135787 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6456f767bd-zpnmp"] Jul 16 21:46:04.139246 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.139226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.142409 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.142378 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 21:46:04.142409 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.142407 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 21:46:04.142656 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.142383 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 21:46:04.142656 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.142488 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 21:46:04.143467 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.143454 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rq4nh\"" Jul 16 21:46:04.143550 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.143477 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 21:46:04.143861 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.143845 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 21:46:04.150265 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.150242 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 21:46:04.159211 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.159183 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 21:46:04.180000 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.179967 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6456f767bd-zpnmp"] Jul 16 21:46:04.234846 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.234756 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:04.292874 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.292836 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-service-ca\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.293074 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.292885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnd42\" (UniqueName: \"kubernetes.io/projected/09d0e255-5d9b-4928-be38-1c5331e8fc11-kube-api-access-fnd42\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.293074 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.292978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-serving-cert\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.293074 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.293019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-oauth-config\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.293074 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.293041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-trusted-ca-bundle\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.293271 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.293102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-oauth-serving-cert\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.293271 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.293145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-config\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.338090 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.338059 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:04.393858 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.393816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-serving-cert\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394034 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.393867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-oauth-config\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394034 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.393888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-trusted-ca-bundle\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394034 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.393935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-oauth-serving-cert\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394034 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.393973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-config\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394218 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.394189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-service-ca\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394274 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.394246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnd42\" (UniqueName: \"kubernetes.io/projected/09d0e255-5d9b-4928-be38-1c5331e8fc11-kube-api-access-fnd42\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394839 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.394808 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-config\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394839 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.394825 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-oauth-serving-cert\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394992 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.394847 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-service-ca\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.394992 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.394897 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09d0e255-5d9b-4928-be38-1c5331e8fc11-trusted-ca-bundle\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.396342 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.396325 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-oauth-config\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.396514 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.396499 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09d0e255-5d9b-4928-be38-1c5331e8fc11-console-serving-cert\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.406180 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.406150 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnd42\" (UniqueName: \"kubernetes.io/projected/09d0e255-5d9b-4928-be38-1c5331e8fc11-kube-api-access-fnd42\") pod \"console-6456f767bd-zpnmp\" (UID: \"09d0e255-5d9b-4928-be38-1c5331e8fc11\") " pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.449212 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.449175 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:04.601623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:04.601598 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6456f767bd-zpnmp"] Jul 16 21:46:04.603988 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:46:04.603957 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d0e255_5d9b_4928_be38_1c5331e8fc11.slice/crio-0dacf9205e70c246789eaf901e4c2e1ff9903d0c39f4bcc990a0a2909c815446 WatchSource:0}: Error finding container 0dacf9205e70c246789eaf901e4c2e1ff9903d0c39f4bcc990a0a2909c815446: Status 404 returned error can't find the container with id 0dacf9205e70c246789eaf901e4c2e1ff9903d0c39f4bcc990a0a2909c815446 Jul 16 21:46:05.573820 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:05.573785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6456f767bd-zpnmp" event={"ID":"09d0e255-5d9b-4928-be38-1c5331e8fc11","Type":"ContainerStarted","Data":"bc1edef4d88df07b3c22349da4988546b0188f6528c558ec3b9866779c6dbc7c"} Jul 16 21:46:05.574253 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:05.573837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6456f767bd-zpnmp" event={"ID":"09d0e255-5d9b-4928-be38-1c5331e8fc11","Type":"ContainerStarted","Data":"0dacf9205e70c246789eaf901e4c2e1ff9903d0c39f4bcc990a0a2909c815446"} Jul 16 21:46:05.594680 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:05.594630 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6456f767bd-zpnmp" podStartSLOduration=1.5946146589999999 podStartE2EDuration="1.594614659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:46:05.592683608 +0000 UTC m=+412.634907871" watchObservedRunningTime="2026-07-16 21:46:05.594614659 +0000 UTC m=+412.636838947" Jul 16 21:46:06.715200 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:06.715166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:06.961738 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:06.961700 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:09.181607 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:09.181573 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:09.308111 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:09.308073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:11.926374 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:11.926328 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:13.089403 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.089364 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms"] Jul 16 21:46:13.093227 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.093203 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:13.096114 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.096087 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 21:46:13.096114 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.096100 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 21:46:13.096304 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.096124 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mvd4n\"" Jul 16 21:46:13.107170 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.107141 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms"] Jul 16 21:46:13.280793 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.280752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzx5\" (UniqueName: \"kubernetes.io/projected/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-kube-api-access-plzx5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" (UID: \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:13.280998 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.280850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" (UID: \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:13.382312 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.382209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" (UID: \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:13.382480 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.382320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzx5\" (UniqueName: \"kubernetes.io/projected/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-kube-api-access-plzx5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" (UID: \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:13.382640 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.382620 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" (UID: \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:13.407440 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.407399 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 21:46:13.415680 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.415645 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 21:46:13.425538 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.425496 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzx5\" (UniqueName: \"kubernetes.io/projected/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-kube-api-access-plzx5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" (UID: \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:13.706424 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.706337 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mvd4n\"" Jul 16 21:46:13.714578 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.714553 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:13.862818 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:13.862794 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms"] Jul 16 21:46:13.867207 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:46:13.867162 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189d0d5c_08af_4a8f_9b72_1abd9653bb0a.slice/crio-173db81128740fa09f19f34bfabc621153449936b38505c05c459ffcb2450c99 WatchSource:0}: Error finding container 173db81128740fa09f19f34bfabc621153449936b38505c05c459ffcb2450c99: Status 404 returned error can't find the container with id 173db81128740fa09f19f34bfabc621153449936b38505c05c459ffcb2450c99 Jul 16 21:46:14.321038 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:14.321003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:14.421473 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:14.421437 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:14.450253 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:14.450219 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:14.450427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:14.450306 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:14.455749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:14.455722 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:14.619202 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:14.619108 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" event={"ID":"189d0d5c-08af-4a8f-9b72-1abd9653bb0a","Type":"ContainerStarted","Data":"173db81128740fa09f19f34bfabc621153449936b38505c05c459ffcb2450c99"} Jul 16 21:46:14.624351 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:14.624282 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6456f767bd-zpnmp" Jul 16 21:46:15.892609 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:15.892566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:15.992892 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:15.992853 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:18.122651 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:18.122612 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:18.192128 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:18.192096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:18.346853 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:18.346816 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:19.560177 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:19.560144 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:21.654206 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:21.654164 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" event={"ID":"189d0d5c-08af-4a8f-9b72-1abd9653bb0a","Type":"ContainerStarted","Data":"8f7e4186f0df6520a85c571a65b21ffcc419d7c28d17a37629dceeaee343f49b"} Jul 16 21:46:21.654659 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:21.654354 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:21.675202 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:21.675147 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" podStartSLOduration=1.9062642429999999 podStartE2EDuration="8.675131707s" totalImagesPullingTime="6.768867464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:46:21.674068342 +0000 UTC m=+428.716292582" watchObservedRunningTime="2026-07-16 21:46:21.675131707 +0000 UTC m=+428.717355945" Jul 16 21:46:22.315278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:22.315242 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:22.416350 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:22.416313 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:23.475123 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:23.475077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:23.574511 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:23.574469 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:25.934797 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:25.934756 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:26.039866 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:26.039826 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:32.661253 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:32.661220 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:33.816045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:33.816007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:33.911460 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:33.911420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:34.102396 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.102274 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:34.188020 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.187982 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:34.348144 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.348110 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:34.447275 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.447181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:34.578052 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.578016 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms"] Jul 16 21:46:34.578265 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.578245 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" podUID="189d0d5c-08af-4a8f-9b72-1abd9653bb0a" containerName="manager" containerID="cri-o://8f7e4186f0df6520a85c571a65b21ffcc419d7c28d17a37629dceeaee343f49b" gracePeriod=2 Jul 16 21:46:34.590172 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.590142 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms"] Jul 16 21:46:34.599390 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:46:34.599349 2573 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\": pods \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:34.608657 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.608630 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r"] Jul 16 21:46:34.609132 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.609112 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189d0d5c-08af-4a8f-9b72-1abd9653bb0a" containerName="manager" Jul 16 21:46:34.609132 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.609135 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="189d0d5c-08af-4a8f-9b72-1abd9653bb0a" containerName="manager" Jul 16 21:46:34.609260 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.609209 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="189d0d5c-08af-4a8f-9b72-1abd9653bb0a" containerName="manager" Jul 16 21:46:34.612407 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.612387 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:34.614846 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:46:34.614778 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="189d0d5c-08af-4a8f-9b72-1abd9653bb0a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:34.630141 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.630109 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r"] Jul 16 21:46:34.666209 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.666182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxsg5\" (UniqueName: \"kubernetes.io/projected/50363090-b551-4c79-be59-f4b2e4b6c019-kube-api-access-sxsg5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wks9r\" (UID: \"50363090-b551-4c79-be59-f4b2e4b6c019\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:34.666209 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.666215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50363090-b551-4c79-be59-f4b2e4b6c019-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wks9r\" (UID: \"50363090-b551-4c79-be59-f4b2e4b6c019\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:34.723684 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.723568 2573 generic.go:356] "Generic (PLEG): container finished" podID="189d0d5c-08af-4a8f-9b72-1abd9653bb0a" containerID="8f7e4186f0df6520a85c571a65b21ffcc419d7c28d17a37629dceeaee343f49b" exitCode=0 Jul 16 21:46:34.767101 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.767068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxsg5\" (UniqueName: \"kubernetes.io/projected/50363090-b551-4c79-be59-f4b2e4b6c019-kube-api-access-sxsg5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wks9r\" (UID: \"50363090-b551-4c79-be59-f4b2e4b6c019\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:34.767278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.767112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50363090-b551-4c79-be59-f4b2e4b6c019-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wks9r\" (UID: \"50363090-b551-4c79-be59-f4b2e4b6c019\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:34.767674 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.767648 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50363090-b551-4c79-be59-f4b2e4b6c019-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wks9r\" (UID: \"50363090-b551-4c79-be59-f4b2e4b6c019\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:34.780406 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.780378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxsg5\" (UniqueName: \"kubernetes.io/projected/50363090-b551-4c79-be59-f4b2e4b6c019-kube-api-access-sxsg5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wks9r\" (UID: \"50363090-b551-4c79-be59-f4b2e4b6c019\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:34.824550 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.824499 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:34.829338 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:46:34.829308 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jk6ms\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="189d0d5c-08af-4a8f-9b72-1abd9653bb0a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:34.867773 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.867733 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plzx5\" (UniqueName: \"kubernetes.io/projected/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-kube-api-access-plzx5\") pod \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\" (UID: \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\") " Jul 16 21:46:34.867941 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.867783 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-extensions-socket-volume\") pod \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\" (UID: \"189d0d5c-08af-4a8f-9b72-1abd9653bb0a\") " Jul 16 21:46:34.868229 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.868207 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "189d0d5c-08af-4a8f-9b72-1abd9653bb0a" (UID: "189d0d5c-08af-4a8f-9b72-1abd9653bb0a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:46:34.869955 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.869937 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-kube-api-access-plzx5" (OuterVolumeSpecName: "kube-api-access-plzx5") pod "189d0d5c-08af-4a8f-9b72-1abd9653bb0a" (UID: "189d0d5c-08af-4a8f-9b72-1abd9653bb0a"). InnerVolumeSpecName "kube-api-access-plzx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:46:34.954925 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.954892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:34.969436 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.969404 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plzx5\" (UniqueName: \"kubernetes.io/projected/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-kube-api-access-plzx5\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:46:34.969436 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:34.969443 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/189d0d5c-08af-4a8f-9b72-1abd9653bb0a-extensions-socket-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:46:35.104292 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:35.104246 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r"] Jul 16 21:46:35.107743 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:46:35.107719 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50363090_b551_4c79_be59_f4b2e4b6c019.slice/crio-f80d40f4682d271e8cc21b89193e96bb3009747cac4e9e64813133062cc00844 WatchSource:0}: Error finding container f80d40f4682d271e8cc21b89193e96bb3009747cac4e9e64813133062cc00844: Status 404 returned error can't find the container with id f80d40f4682d271e8cc21b89193e96bb3009747cac4e9e64813133062cc00844 Jul 16 21:46:35.540587 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:35.540556 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="189d0d5c-08af-4a8f-9b72-1abd9653bb0a" path="/var/lib/kubelet/pods/189d0d5c-08af-4a8f-9b72-1abd9653bb0a/volumes" Jul 16 21:46:35.731203 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:35.731161 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" event={"ID":"50363090-b551-4c79-be59-f4b2e4b6c019","Type":"ContainerStarted","Data":"b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7"} Jul 16 21:46:35.731414 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:35.731231 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" event={"ID":"50363090-b551-4c79-be59-f4b2e4b6c019","Type":"ContainerStarted","Data":"f80d40f4682d271e8cc21b89193e96bb3009747cac4e9e64813133062cc00844"} Jul 16 21:46:35.731481 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:35.731434 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:46:35.733034 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:35.733015 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jk6ms" Jul 16 21:46:35.733173 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:35.733063 2573 scope.go:117] "RemoveContainer" containerID="8f7e4186f0df6520a85c571a65b21ffcc419d7c28d17a37629dceeaee343f49b" Jul 16 21:46:35.751090 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:35.751035 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" podStartSLOduration=1.7510189390000002 podStartE2EDuration="1.751018939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:46:35.749761866 +0000 UTC m=+442.791986100" watchObservedRunningTime="2026-07-16 21:46:35.751018939 +0000 UTC m=+442.793243178" Jul 16 21:46:36.738970 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:36.738925 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:36.837138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:36.837104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:41.926143 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:41.926104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:42.026312 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:42.026271 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:44.486837 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:44.486802 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:44.588372 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:44.588334 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:46:46.741259 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:46:46.741230 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:47:01.834054 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:01.834020 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r"] Jul 16 21:47:01.834462 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:01.834221 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" podUID="50363090-b551-4c79-be59-f4b2e4b6c019" containerName="manager" containerID="cri-o://b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7" gracePeriod=10 Jul 16 21:47:02.035120 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.035087 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw"] Jul 16 21:47:02.037561 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.037541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.056939 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.056907 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw"] Jul 16 21:47:02.089361 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.089290 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:47:02.130110 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.130080 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxsg5\" (UniqueName: \"kubernetes.io/projected/50363090-b551-4c79-be59-f4b2e4b6c019-kube-api-access-sxsg5\") pod \"50363090-b551-4c79-be59-f4b2e4b6c019\" (UID: \"50363090-b551-4c79-be59-f4b2e4b6c019\") " Jul 16 21:47:02.130292 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.130176 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50363090-b551-4c79-be59-f4b2e4b6c019-extensions-socket-volume\") pod \"50363090-b551-4c79-be59-f4b2e4b6c019\" (UID: \"50363090-b551-4c79-be59-f4b2e4b6c019\") " Jul 16 21:47:02.130358 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.130337 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468p2\" (UniqueName: \"kubernetes.io/projected/3cb80963-c28c-41c7-b0ed-73bca4d633e8-kube-api-access-468p2\") pod \"kuadrant-operator-controller-manager-86878c769f-4nslw\" (UID: \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.130404 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.130376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb80963-c28c-41c7-b0ed-73bca4d633e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4nslw\" (UID: \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.130709 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.130671 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50363090-b551-4c79-be59-f4b2e4b6c019-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "50363090-b551-4c79-be59-f4b2e4b6c019" (UID: "50363090-b551-4c79-be59-f4b2e4b6c019"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:47:02.132331 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.132304 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50363090-b551-4c79-be59-f4b2e4b6c019-kube-api-access-sxsg5" (OuterVolumeSpecName: "kube-api-access-sxsg5") pod "50363090-b551-4c79-be59-f4b2e4b6c019" (UID: "50363090-b551-4c79-be59-f4b2e4b6c019"). InnerVolumeSpecName "kube-api-access-sxsg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:47:02.230915 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.230868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468p2\" (UniqueName: \"kubernetes.io/projected/3cb80963-c28c-41c7-b0ed-73bca4d633e8-kube-api-access-468p2\") pod \"kuadrant-operator-controller-manager-86878c769f-4nslw\" (UID: \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.230915 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.230916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb80963-c28c-41c7-b0ed-73bca4d633e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4nslw\" (UID: \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.231132 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.230988 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxsg5\" (UniqueName: \"kubernetes.io/projected/50363090-b551-4c79-be59-f4b2e4b6c019-kube-api-access-sxsg5\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:47:02.231132 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.230998 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50363090-b551-4c79-be59-f4b2e4b6c019-extensions-socket-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:47:02.231308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.231292 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb80963-c28c-41c7-b0ed-73bca4d633e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4nslw\" (UID: \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.240145 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.240108 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-468p2\" (UniqueName: \"kubernetes.io/projected/3cb80963-c28c-41c7-b0ed-73bca4d633e8-kube-api-access-468p2\") pod \"kuadrant-operator-controller-manager-86878c769f-4nslw\" (UID: \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.348433 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.348329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.490576 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.490548 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw"] Jul 16 21:47:02.492921 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:47:02.492898 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb80963_c28c_41c7_b0ed_73bca4d633e8.slice/crio-8d941af126ffc77042811dbf30ff1d3db8c25a88822ddb04857373e44cd564e5 WatchSource:0}: Error finding container 8d941af126ffc77042811dbf30ff1d3db8c25a88822ddb04857373e44cd564e5: Status 404 returned error can't find the container with id 8d941af126ffc77042811dbf30ff1d3db8c25a88822ddb04857373e44cd564e5 Jul 16 21:47:02.870707 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.870671 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" event={"ID":"3cb80963-c28c-41c7-b0ed-73bca4d633e8","Type":"ContainerStarted","Data":"d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c"} Jul 16 21:47:02.871215 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.870764 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:02.871215 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.870786 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" event={"ID":"3cb80963-c28c-41c7-b0ed-73bca4d633e8","Type":"ContainerStarted","Data":"8d941af126ffc77042811dbf30ff1d3db8c25a88822ddb04857373e44cd564e5"} Jul 16 21:47:02.872093 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.872069 2573 generic.go:356] "Generic (PLEG): container finished" podID="50363090-b551-4c79-be59-f4b2e4b6c019" containerID="b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7" exitCode=0 Jul 16 21:47:02.872198 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.872113 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" Jul 16 21:47:02.872198 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.872137 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" event={"ID":"50363090-b551-4c79-be59-f4b2e4b6c019","Type":"ContainerDied","Data":"b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7"} Jul 16 21:47:02.872198 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.872172 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r" event={"ID":"50363090-b551-4c79-be59-f4b2e4b6c019","Type":"ContainerDied","Data":"f80d40f4682d271e8cc21b89193e96bb3009747cac4e9e64813133062cc00844"} Jul 16 21:47:02.872198 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.872193 2573 scope.go:117] "RemoveContainer" containerID="b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7" Jul 16 21:47:02.881366 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.881342 2573 scope.go:117] "RemoveContainer" containerID="b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7" Jul 16 21:47:02.881683 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:47:02.881664 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7\": container with ID starting with b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7 not found: ID does not exist" containerID="b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7" Jul 16 21:47:02.881746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.881703 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7"} err="failed to get container status \"b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7\": rpc error: code = NotFound desc = could not find container \"b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7\": container with ID starting with b27ad605959a842533337f58a390e0385c2487d22155c8fec5d0660f6524aef7 not found: ID does not exist" Jul 16 21:47:02.890752 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.890700 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" podStartSLOduration=0.890684613 podStartE2EDuration="890.684613ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:02.888762574 +0000 UTC m=+469.930986813" watchObservedRunningTime="2026-07-16 21:47:02.890684613 +0000 UTC m=+469.932908851" Jul 16 21:47:02.903463 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.903427 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r"] Jul 16 21:47:02.907056 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:02.907025 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wks9r"] Jul 16 21:47:03.540880 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:03.540847 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50363090-b551-4c79-be59-f4b2e4b6c019" path="/var/lib/kubelet/pods/50363090-b551-4c79-be59-f4b2e4b6c019/volumes" Jul 16 21:47:03.883557 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:03.883512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:03.983777 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:03.983742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:13.360230 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:13.360195 2573 scope.go:117] "RemoveContainer" containerID="1ee6192f4579d815136be37f9d535dd36bb368326464a62cadc65dcfad3ba714" Jul 16 21:47:13.881058 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:13.881026 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:47:18.276100 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:18.276062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:18.784478 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:18.784442 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:20.486818 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:20.486761 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:20.607586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:20.607553 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:20.669672 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:20.669633 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:21.321320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.321279 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-jdj5z"] Jul 16 21:47:21.321730 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.321714 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50363090-b551-4c79-be59-f4b2e4b6c019" containerName="manager" Jul 16 21:47:21.321792 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.321733 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="50363090-b551-4c79-be59-f4b2e4b6c019" containerName="manager" Jul 16 21:47:21.321829 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.321798 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="50363090-b551-4c79-be59-f4b2e4b6c019" containerName="manager" Jul 16 21:47:21.324461 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.324443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jdj5z" Jul 16 21:47:21.327515 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.327495 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x8t5s\"" Jul 16 21:47:21.334719 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.334692 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jdj5z"] Jul 16 21:47:21.366771 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.366736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:21.419976 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.419931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnmfm\" (UniqueName: \"kubernetes.io/projected/023360eb-75bd-41ee-b2bd-0ca78b7bdb07-kube-api-access-gnmfm\") pod \"authorino-58cd955675-jdj5z\" (UID: \"023360eb-75bd-41ee-b2bd-0ca78b7bdb07\") " pod="kuadrant-system/authorino-58cd955675-jdj5z" Jul 16 21:47:21.467255 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.467219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:21.521370 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.521329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnmfm\" (UniqueName: \"kubernetes.io/projected/023360eb-75bd-41ee-b2bd-0ca78b7bdb07-kube-api-access-gnmfm\") pod \"authorino-58cd955675-jdj5z\" (UID: \"023360eb-75bd-41ee-b2bd-0ca78b7bdb07\") " pod="kuadrant-system/authorino-58cd955675-jdj5z" Jul 16 21:47:21.532532 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.532487 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnmfm\" (UniqueName: \"kubernetes.io/projected/023360eb-75bd-41ee-b2bd-0ca78b7bdb07-kube-api-access-gnmfm\") pod \"authorino-58cd955675-jdj5z\" (UID: \"023360eb-75bd-41ee-b2bd-0ca78b7bdb07\") " pod="kuadrant-system/authorino-58cd955675-jdj5z" Jul 16 21:47:21.634570 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.634449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jdj5z" Jul 16 21:47:21.636980 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.636724 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-kthjc"] Jul 16 21:47:21.639320 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.639298 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kthjc" Jul 16 21:47:21.660749 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.660680 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kthjc"] Jul 16 21:47:21.723408 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.723365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfwn\" (UniqueName: \"kubernetes.io/projected/e8299093-92db-4c1e-9463-638d7d2bd7d8-kube-api-access-6tfwn\") pod \"authorino-75d556bd77-kthjc\" (UID: \"e8299093-92db-4c1e-9463-638d7d2bd7d8\") " pod="kuadrant-system/authorino-75d556bd77-kthjc" Jul 16 21:47:21.786091 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.786065 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jdj5z"] Jul 16 21:47:21.788934 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:47:21.788907 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod023360eb_75bd_41ee_b2bd_0ca78b7bdb07.slice/crio-8856c046e2b90b4fcf779f5880b55167f4d853561b3990398251eca3478d1fa0 WatchSource:0}: Error finding container 8856c046e2b90b4fcf779f5880b55167f4d853561b3990398251eca3478d1fa0: Status 404 returned error can't find the container with id 8856c046e2b90b4fcf779f5880b55167f4d853561b3990398251eca3478d1fa0 Jul 16 21:47:21.824119 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.824088 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfwn\" (UniqueName: \"kubernetes.io/projected/e8299093-92db-4c1e-9463-638d7d2bd7d8-kube-api-access-6tfwn\") pod \"authorino-75d556bd77-kthjc\" (UID: \"e8299093-92db-4c1e-9463-638d7d2bd7d8\") " pod="kuadrant-system/authorino-75d556bd77-kthjc" Jul 16 21:47:21.840164 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.840131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfwn\" (UniqueName: \"kubernetes.io/projected/e8299093-92db-4c1e-9463-638d7d2bd7d8-kube-api-access-6tfwn\") pod \"authorino-75d556bd77-kthjc\" (UID: \"e8299093-92db-4c1e-9463-638d7d2bd7d8\") " pod="kuadrant-system/authorino-75d556bd77-kthjc" Jul 16 21:47:21.965817 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.965697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kthjc" Jul 16 21:47:21.968098 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:21.968068 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jdj5z" event={"ID":"023360eb-75bd-41ee-b2bd-0ca78b7bdb07","Type":"ContainerStarted","Data":"8856c046e2b90b4fcf779f5880b55167f4d853561b3990398251eca3478d1fa0"} Jul 16 21:47:22.107255 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:22.107231 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kthjc"] Jul 16 21:47:22.109580 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:47:22.109554 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8299093_92db_4c1e_9463_638d7d2bd7d8.slice/crio-238a4d61a2280cab9753f9be37e006155ec3fdb50ca259652fcc290e6a1cc125 WatchSource:0}: Error finding container 238a4d61a2280cab9753f9be37e006155ec3fdb50ca259652fcc290e6a1cc125: Status 404 returned error can't find the container with id 238a4d61a2280cab9753f9be37e006155ec3fdb50ca259652fcc290e6a1cc125 Jul 16 21:47:22.982877 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:22.982838 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kthjc" event={"ID":"e8299093-92db-4c1e-9463-638d7d2bd7d8","Type":"ContainerStarted","Data":"238a4d61a2280cab9753f9be37e006155ec3fdb50ca259652fcc290e6a1cc125"} Jul 16 21:47:26.010178 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:26.010133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jdj5z" event={"ID":"023360eb-75bd-41ee-b2bd-0ca78b7bdb07","Type":"ContainerStarted","Data":"179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279"} Jul 16 21:47:26.011784 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:26.011758 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kthjc" event={"ID":"e8299093-92db-4c1e-9463-638d7d2bd7d8","Type":"ContainerStarted","Data":"5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4"} Jul 16 21:47:26.027304 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:26.027251 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-jdj5z" podStartSLOduration=1.463532565 podStartE2EDuration="5.027235362s" totalImagesPullingTime="3.563702797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:26.026054787 +0000 UTC m=+493.068279038" watchObservedRunningTime="2026-07-16 21:47:26.027235362 +0000 UTC m=+493.069459674" Jul 16 21:47:26.044409 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:26.044355 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-kthjc" podStartSLOduration=1.8097207499999999 podStartE2EDuration="5.044336583s" totalImagesPullingTime="3.234615833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:26.042480024 +0000 UTC m=+493.084704262" watchObservedRunningTime="2026-07-16 21:47:26.044336583 +0000 UTC m=+493.086560821" Jul 16 21:47:26.070154 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:26.070117 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jdj5z"] Jul 16 21:47:28.021879 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:28.021838 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-jdj5z" podUID="023360eb-75bd-41ee-b2bd-0ca78b7bdb07" containerName="authorino" containerID="cri-o://179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279" gracePeriod=30 Jul 16 21:47:28.273080 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:28.273002 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jdj5z" Jul 16 21:47:28.389466 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:28.389431 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnmfm\" (UniqueName: \"kubernetes.io/projected/023360eb-75bd-41ee-b2bd-0ca78b7bdb07-kube-api-access-gnmfm\") pod \"023360eb-75bd-41ee-b2bd-0ca78b7bdb07\" (UID: \"023360eb-75bd-41ee-b2bd-0ca78b7bdb07\") " Jul 16 21:47:28.391563 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:28.391514 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023360eb-75bd-41ee-b2bd-0ca78b7bdb07-kube-api-access-gnmfm" (OuterVolumeSpecName: "kube-api-access-gnmfm") pod "023360eb-75bd-41ee-b2bd-0ca78b7bdb07" (UID: "023360eb-75bd-41ee-b2bd-0ca78b7bdb07"). InnerVolumeSpecName "kube-api-access-gnmfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:47:28.490112 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:28.490079 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnmfm\" (UniqueName: \"kubernetes.io/projected/023360eb-75bd-41ee-b2bd-0ca78b7bdb07-kube-api-access-gnmfm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:47:29.028298 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.028255 2573 generic.go:356] "Generic (PLEG): container finished" podID="023360eb-75bd-41ee-b2bd-0ca78b7bdb07" containerID="179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279" exitCode=0 Jul 16 21:47:29.028298 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.028303 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jdj5z" Jul 16 21:47:29.028806 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.028325 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jdj5z" event={"ID":"023360eb-75bd-41ee-b2bd-0ca78b7bdb07","Type":"ContainerDied","Data":"179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279"} Jul 16 21:47:29.028806 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.028364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jdj5z" event={"ID":"023360eb-75bd-41ee-b2bd-0ca78b7bdb07","Type":"ContainerDied","Data":"8856c046e2b90b4fcf779f5880b55167f4d853561b3990398251eca3478d1fa0"} Jul 16 21:47:29.028806 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.028381 2573 scope.go:117] "RemoveContainer" containerID="179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279" Jul 16 21:47:29.042569 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.042544 2573 scope.go:117] "RemoveContainer" containerID="179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279" Jul 16 21:47:29.042881 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:47:29.042859 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279\": container with ID starting with 179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279 not found: ID does not exist" containerID="179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279" Jul 16 21:47:29.042927 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.042892 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279"} err="failed to get container status \"179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279\": rpc error: code = NotFound desc = could not find container \"179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279\": container with ID starting with 179eb63f2510e1bff3e5924d77d74611f4b85a2408ed9ead80845806c0dc4279 not found: ID does not exist" Jul 16 21:47:29.056166 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.056128 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jdj5z"] Jul 16 21:47:29.063632 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.063603 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-jdj5z"] Jul 16 21:47:29.541292 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:29.541258 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023360eb-75bd-41ee-b2bd-0ca78b7bdb07" path="/var/lib/kubelet/pods/023360eb-75bd-41ee-b2bd-0ca78b7bdb07/volumes" Jul 16 21:47:41.118265 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:41.118231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:41.216191 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:41.216146 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:41.557840 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:41.557807 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:53.404227 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.404128 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:53.500995 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.500960 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:53.844465 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.844424 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-zdk2k"] Jul 16 21:47:53.844873 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.844860 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023360eb-75bd-41ee-b2bd-0ca78b7bdb07" containerName="authorino" Jul 16 21:47:53.844922 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.844875 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="023360eb-75bd-41ee-b2bd-0ca78b7bdb07" containerName="authorino" Jul 16 21:47:53.844957 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.844938 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="023360eb-75bd-41ee-b2bd-0ca78b7bdb07" containerName="authorino" Jul 16 21:47:53.849159 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.849127 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:53.851617 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.851589 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 21:47:53.857840 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.857803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-zdk2k"] Jul 16 21:47:53.901708 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.901668 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-zdk2k"] Jul 16 21:47:53.902018 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:47:53.901993 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cnzw6 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-zdk2k" podUID="93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1" Jul 16 21:47:53.927359 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.927324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-tls-cert\") pod \"authorino-76877994-zdk2k\" (UID: \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\") " pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:53.927359 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.927371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzw6\" (UniqueName: \"kubernetes.io/projected/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-kube-api-access-cnzw6\") pod \"authorino-76877994-zdk2k\" (UID: \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\") " pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:53.933996 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.933963 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64b9597d5f-9mhgs"] Jul 16 21:47:53.936782 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.936760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:47:53.945050 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:53.945021 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64b9597d5f-9mhgs"] Jul 16 21:47:54.028780 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.028718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-tls-cert\") pod \"authorino-76877994-zdk2k\" (UID: \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\") " pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:54.028780 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.028774 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1400e9e9-978b-4564-9f4b-67ad29ca29e3-tls-cert\") pod \"authorino-64b9597d5f-9mhgs\" (UID: \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\") " pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:47:54.028780 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.028799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzw6\" (UniqueName: \"kubernetes.io/projected/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-kube-api-access-cnzw6\") pod \"authorino-76877994-zdk2k\" (UID: \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\") " pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:54.029067 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.028860 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxct5\" (UniqueName: \"kubernetes.io/projected/1400e9e9-978b-4564-9f4b-67ad29ca29e3-kube-api-access-rxct5\") pod \"authorino-64b9597d5f-9mhgs\" (UID: \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\") " pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:47:54.031209 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.031183 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-tls-cert\") pod \"authorino-76877994-zdk2k\" (UID: \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\") " pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:54.036896 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.036869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzw6\" (UniqueName: \"kubernetes.io/projected/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-kube-api-access-cnzw6\") pod \"authorino-76877994-zdk2k\" (UID: \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\") " pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:54.130343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.130244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1400e9e9-978b-4564-9f4b-67ad29ca29e3-tls-cert\") pod \"authorino-64b9597d5f-9mhgs\" (UID: \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\") " pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:47:54.130343 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.130296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxct5\" (UniqueName: \"kubernetes.io/projected/1400e9e9-978b-4564-9f4b-67ad29ca29e3-kube-api-access-rxct5\") pod \"authorino-64b9597d5f-9mhgs\" (UID: \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\") " pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:47:54.132949 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.132919 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1400e9e9-978b-4564-9f4b-67ad29ca29e3-tls-cert\") pod \"authorino-64b9597d5f-9mhgs\" (UID: \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\") " pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:47:54.139838 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.139809 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxct5\" (UniqueName: \"kubernetes.io/projected/1400e9e9-978b-4564-9f4b-67ad29ca29e3-kube-api-access-rxct5\") pod \"authorino-64b9597d5f-9mhgs\" (UID: \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\") " pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:47:54.151042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.151011 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:54.157715 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.157688 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:54.231087 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.231047 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnzw6\" (UniqueName: \"kubernetes.io/projected/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-kube-api-access-cnzw6\") pod \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\" (UID: \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\") " Jul 16 21:47:54.231289 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.231161 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-tls-cert\") pod \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\" (UID: \"93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1\") " Jul 16 21:47:54.233196 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.233158 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1" (UID: "93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:47:54.233350 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.233303 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-kube-api-access-cnzw6" (OuterVolumeSpecName: "kube-api-access-cnzw6") pod "93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1" (UID: "93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1"). InnerVolumeSpecName "kube-api-access-cnzw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:47:54.246472 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.246441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:47:54.332126 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.331956 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzw6\" (UniqueName: \"kubernetes.io/projected/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-kube-api-access-cnzw6\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:47:54.332126 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.331978 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1-tls-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:47:54.381627 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:54.381607 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64b9597d5f-9mhgs"] Jul 16 21:47:54.383852 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:47:54.383830 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1400e9e9_978b_4564_9f4b_67ad29ca29e3.slice/crio-ca20aca4b7a850387976cb83b612a3497fb2753a16c17713c3cde8054521e0ed WatchSource:0}: Error finding container ca20aca4b7a850387976cb83b612a3497fb2753a16c17713c3cde8054521e0ed: Status 404 returned error can't find the container with id ca20aca4b7a850387976cb83b612a3497fb2753a16c17713c3cde8054521e0ed Jul 16 21:47:55.157586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.157450 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" event={"ID":"1400e9e9-978b-4564-9f4b-67ad29ca29e3","Type":"ContainerStarted","Data":"bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6"} Jul 16 21:47:55.157586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.157497 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" event={"ID":"1400e9e9-978b-4564-9f4b-67ad29ca29e3","Type":"ContainerStarted","Data":"ca20aca4b7a850387976cb83b612a3497fb2753a16c17713c3cde8054521e0ed"} Jul 16 21:47:55.157586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.157465 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-zdk2k" Jul 16 21:47:55.173043 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.172990 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" podStartSLOduration=1.728641645 podStartE2EDuration="2.172969881s" totalImagesPullingTime="444.328236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:55.171989126 +0000 UTC m=+522.214213365" watchObservedRunningTime="2026-07-16 21:47:55.172969881 +0000 UTC m=+522.215194140" Jul 16 21:47:55.205604 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.205569 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-zdk2k"] Jul 16 21:47:55.207676 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.207629 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-zdk2k"] Jul 16 21:47:55.210412 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.210382 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kthjc"] Jul 16 21:47:55.210624 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.210599 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-kthjc" podUID="e8299093-92db-4c1e-9463-638d7d2bd7d8" containerName="authorino" containerID="cri-o://5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4" gracePeriod=30 Jul 16 21:47:55.466588 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.466562 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kthjc" Jul 16 21:47:55.541546 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.541486 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1" path="/var/lib/kubelet/pods/93d3a7d8-5a7f-4174-9b13-0f0cf1505bc1/volumes" Jul 16 21:47:55.543487 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.543462 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tfwn\" (UniqueName: \"kubernetes.io/projected/e8299093-92db-4c1e-9463-638d7d2bd7d8-kube-api-access-6tfwn\") pod \"e8299093-92db-4c1e-9463-638d7d2bd7d8\" (UID: \"e8299093-92db-4c1e-9463-638d7d2bd7d8\") " Jul 16 21:47:55.545475 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.545444 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8299093-92db-4c1e-9463-638d7d2bd7d8-kube-api-access-6tfwn" (OuterVolumeSpecName: "kube-api-access-6tfwn") pod "e8299093-92db-4c1e-9463-638d7d2bd7d8" (UID: "e8299093-92db-4c1e-9463-638d7d2bd7d8"). InnerVolumeSpecName "kube-api-access-6tfwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:47:55.645126 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:55.645087 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfwn\" (UniqueName: \"kubernetes.io/projected/e8299093-92db-4c1e-9463-638d7d2bd7d8-kube-api-access-6tfwn\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:47:56.166820 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.166787 2573 generic.go:356] "Generic (PLEG): container finished" podID="e8299093-92db-4c1e-9463-638d7d2bd7d8" containerID="5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4" exitCode=0 Jul 16 21:47:56.167222 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.166829 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kthjc" Jul 16 21:47:56.167222 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.166882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kthjc" event={"ID":"e8299093-92db-4c1e-9463-638d7d2bd7d8","Type":"ContainerDied","Data":"5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4"} Jul 16 21:47:56.167222 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.166921 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kthjc" event={"ID":"e8299093-92db-4c1e-9463-638d7d2bd7d8","Type":"ContainerDied","Data":"238a4d61a2280cab9753f9be37e006155ec3fdb50ca259652fcc290e6a1cc125"} Jul 16 21:47:56.167222 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.166941 2573 scope.go:117] "RemoveContainer" containerID="5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4" Jul 16 21:47:56.176283 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.176259 2573 scope.go:117] "RemoveContainer" containerID="5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4" Jul 16 21:47:56.176629 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:47:56.176606 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4\": container with ID starting with 5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4 not found: ID does not exist" containerID="5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4" Jul 16 21:47:56.176731 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.176637 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4"} err="failed to get container status \"5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4\": rpc error: code = NotFound desc = could not find container \"5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4\": container with ID starting with 5e6e06a37c4cf40b693db52ea8beba3082e167288c59ec68aca1d3258e1cf8e4 not found: ID does not exist" Jul 16 21:47:56.187697 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.187663 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kthjc"] Jul 16 21:47:56.189413 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:56.189387 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kthjc"] Jul 16 21:47:57.541373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:57.541334 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8299093-92db-4c1e-9463-638d7d2bd7d8" path="/var/lib/kubelet/pods/e8299093-92db-4c1e-9463-638d7d2bd7d8/volumes" Jul 16 21:47:59.421102 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:59.421071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:59.511504 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:59.511469 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:59.568318 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:59.568282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:59.672331 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:47:59.672239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:02.472014 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:02.471972 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:02.587880 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:02.587850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:34.971258 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:34.971211 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:35.421380 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:35.421343 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:35.781994 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:35.781901 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:36.315516 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.315476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:36.424057 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.424019 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:36.606088 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.606003 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vmmsf"] Jul 16 21:48:36.606387 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.606376 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8299093-92db-4c1e-9463-638d7d2bd7d8" containerName="authorino" Jul 16 21:48:36.606443 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.606389 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8299093-92db-4c1e-9463-638d7d2bd7d8" containerName="authorino" Jul 16 21:48:36.606479 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.606456 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8299093-92db-4c1e-9463-638d7d2bd7d8" containerName="authorino" Jul 16 21:48:36.609769 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.609748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:36.613448 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.613422 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g4pjt\"" Jul 16 21:48:36.613581 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.613554 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 21:48:36.624883 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.624839 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vmmsf"] Jul 16 21:48:36.735099 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.735052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e5398f-c17b-405e-8855-67a8fdbb929c-plugins-config-volume\") pod \"payload-processing-58f86c7798-vmmsf\" (UID: \"a5e5398f-c17b-405e-8855-67a8fdbb929c\") " pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:36.735278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.735247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qn8j\" (UniqueName: \"kubernetes.io/projected/a5e5398f-c17b-405e-8855-67a8fdbb929c-kube-api-access-2qn8j\") pod \"payload-processing-58f86c7798-vmmsf\" (UID: \"a5e5398f-c17b-405e-8855-67a8fdbb929c\") " pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:36.836119 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.836075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qn8j\" (UniqueName: \"kubernetes.io/projected/a5e5398f-c17b-405e-8855-67a8fdbb929c-kube-api-access-2qn8j\") pod \"payload-processing-58f86c7798-vmmsf\" (UID: \"a5e5398f-c17b-405e-8855-67a8fdbb929c\") " pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:36.836119 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.836122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e5398f-c17b-405e-8855-67a8fdbb929c-plugins-config-volume\") pod \"payload-processing-58f86c7798-vmmsf\" (UID: \"a5e5398f-c17b-405e-8855-67a8fdbb929c\") " pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:36.836985 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.836945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e5398f-c17b-405e-8855-67a8fdbb929c-plugins-config-volume\") pod \"payload-processing-58f86c7798-vmmsf\" (UID: \"a5e5398f-c17b-405e-8855-67a8fdbb929c\") " pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:36.846360 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.846328 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qn8j\" (UniqueName: \"kubernetes.io/projected/a5e5398f-c17b-405e-8855-67a8fdbb929c-kube-api-access-2qn8j\") pod \"payload-processing-58f86c7798-vmmsf\" (UID: \"a5e5398f-c17b-405e-8855-67a8fdbb929c\") " pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:36.920687 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:36.920586 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:37.073898 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:37.073874 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vmmsf"] Jul 16 21:48:37.076131 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:48:37.076105 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e5398f_c17b_405e_8855_67a8fdbb929c.slice/crio-5c3657d16679c6b3c6a9f2006fca33b11ef3ba4aa993e54fd37cedc7246e47e4 WatchSource:0}: Error finding container 5c3657d16679c6b3c6a9f2006fca33b11ef3ba4aa993e54fd37cedc7246e47e4: Status 404 returned error can't find the container with id 5c3657d16679c6b3c6a9f2006fca33b11ef3ba4aa993e54fd37cedc7246e47e4 Jul 16 21:48:37.369744 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:37.369697 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" event={"ID":"a5e5398f-c17b-405e-8855-67a8fdbb929c","Type":"ContainerStarted","Data":"5c3657d16679c6b3c6a9f2006fca33b11ef3ba4aa993e54fd37cedc7246e47e4"} Jul 16 21:48:39.381067 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:39.381031 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" event={"ID":"a5e5398f-c17b-405e-8855-67a8fdbb929c","Type":"ContainerStarted","Data":"ee9f9c8c05523a60687cd89c92404f04ea7e77c990c55ce61f72c51fafc0a015"} Jul 16 21:48:39.381591 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:39.381165 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:39.403302 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:39.403245 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" podStartSLOduration=2.175562511 podStartE2EDuration="3.403225712s" totalImagesPullingTime="1.227663201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:39.40139673 +0000 UTC m=+566.443620985" watchObservedRunningTime="2026-07-16 21:48:39.403225712 +0000 UTC m=+566.445449949" Jul 16 21:48:46.323212 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:46.323174 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:46.431087 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:46.431054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:50.388216 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:50.388180 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-vmmsf" Jul 16 21:48:51.023210 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:51.023175 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:52.329719 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:52.329681 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:52.431654 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:48:52.431613 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:01.943990 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:01.943953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:02.069080 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:02.069049 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:19.385331 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.385294 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8"] Jul 16 21:49:19.396014 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.395974 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.398798 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.398768 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8"] Jul 16 21:49:19.399587 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.399557 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9l2xf\"" Jul 16 21:49:19.399716 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.399557 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 16 21:49:19.399716 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.399561 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 21:49:19.399838 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.399563 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 21:49:19.432482 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.432426 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:19.529497 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.529455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23352deb-b450-4b5d-82de-3f5c1e4de8ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.529737 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.529513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhggq\" (UniqueName: \"kubernetes.io/projected/23352deb-b450-4b5d-82de-3f5c1e4de8ee-kube-api-access-mhggq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.529737 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.529614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.529737 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.529670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.529737 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.529708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.529912 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.529797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631003 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.630957 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23352deb-b450-4b5d-82de-3f5c1e4de8ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631182 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.631046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhggq\" (UniqueName: \"kubernetes.io/projected/23352deb-b450-4b5d-82de-3f5c1e4de8ee-kube-api-access-mhggq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631182 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.631074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631182 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.631109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631182 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.631136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631410 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.631215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631617 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.631586 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631756 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.631634 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.631756 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.631674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.633685 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.633659 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23352deb-b450-4b5d-82de-3f5c1e4de8ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.633939 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.633922 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23352deb-b450-4b5d-82de-3f5c1e4de8ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.640683 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.640600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhggq\" (UniqueName: \"kubernetes.io/projected/23352deb-b450-4b5d-82de-3f5c1e4de8ee-kube-api-access-mhggq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wj4s8\" (UID: \"23352deb-b450-4b5d-82de-3f5c1e4de8ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.709697 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.709658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:19.853026 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:19.852985 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8"] Jul 16 21:49:19.855345 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:49:19.855316 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23352deb_b450_4b5d_82de_3f5c1e4de8ee.slice/crio-c0cce32dfc33f1a47e8d08861f66247910c512850400bfbaf78f92edffbcf772 WatchSource:0}: Error finding container c0cce32dfc33f1a47e8d08861f66247910c512850400bfbaf78f92edffbcf772: Status 404 returned error can't find the container with id c0cce32dfc33f1a47e8d08861f66247910c512850400bfbaf78f92edffbcf772 Jul 16 21:49:20.583308 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:20.583269 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" event={"ID":"23352deb-b450-4b5d-82de-3f5c1e4de8ee","Type":"ContainerStarted","Data":"c0cce32dfc33f1a47e8d08861f66247910c512850400bfbaf78f92edffbcf772"} Jul 16 21:49:21.590151 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:21.590116 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" event={"ID":"23352deb-b450-4b5d-82de-3f5c1e4de8ee","Type":"ContainerStarted","Data":"033a72742f1ab530339acb0d72a7ab0523c0afba8f6ec9aeaff9db8ecc7ae5c7"} Jul 16 21:49:21.610599 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:21.610539 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" podStartSLOduration=1.47825122 podStartE2EDuration="2.610508999s" totalImagesPullingTime="1.132257779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:21.60863881 +0000 UTC m=+608.650863061" watchObservedRunningTime="2026-07-16 21:49:21.610508999 +0000 UTC m=+608.652733236" Jul 16 21:49:29.710100 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:29.710056 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:29.710600 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:29.710129 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:29.722628 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:29.722598 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:30.644415 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:30.644388 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wj4s8" Jul 16 21:49:50.972887 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:50.972847 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:54.482015 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.481977 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv"] Jul 16 21:49:54.485750 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.485726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.490537 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.490491 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 16 21:49:54.496279 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.496254 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv"] Jul 16 21:49:54.521493 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.521459 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:54.568154 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.568107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b13a4c5-ceaf-4561-a854-f10942a0369f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.568154 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.568157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pwt\" (UniqueName: \"kubernetes.io/projected/6b13a4c5-ceaf-4561-a854-f10942a0369f-kube-api-access-p7pwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.568422 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.568186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.568422 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.568250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.568422 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.568302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.568422 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.568334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.668970 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.668920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b13a4c5-ceaf-4561-a854-f10942a0369f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.668970 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.668967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pwt\" (UniqueName: \"kubernetes.io/projected/6b13a4c5-ceaf-4561-a854-f10942a0369f-kube-api-access-p7pwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.669243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.668992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.669243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.669021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.669243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.669045 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.669243 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.669068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.669479 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.669449 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.669551 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.669470 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.669597 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.669556 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.671327 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.671305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b13a4c5-ceaf-4561-a854-f10942a0369f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.671489 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.671471 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b13a4c5-ceaf-4561-a854-f10942a0369f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.677373 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.677346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pwt\" (UniqueName: \"kubernetes.io/projected/6b13a4c5-ceaf-4561-a854-f10942a0369f-kube-api-access-p7pwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x4lsv\" (UID: \"6b13a4c5-ceaf-4561-a854-f10942a0369f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.797053 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.797010 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:49:54.937486 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.937454 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv"] Jul 16 21:49:54.939890 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:49:54.939861 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b13a4c5_ceaf_4561_a854_f10942a0369f.slice/crio-11d3dbb60a67ffb1c9cc4a4c7f123b2c62d4010572391001254b4b933ca7643b WatchSource:0}: Error finding container 11d3dbb60a67ffb1c9cc4a4c7f123b2c62d4010572391001254b4b933ca7643b: Status 404 returned error can't find the container with id 11d3dbb60a67ffb1c9cc4a4c7f123b2c62d4010572391001254b4b933ca7643b Jul 16 21:49:54.941633 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:54.941614 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:49:55.756858 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:55.756821 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" event={"ID":"6b13a4c5-ceaf-4561-a854-f10942a0369f","Type":"ContainerStarted","Data":"121967237c964d7a25ac20fc221800b336d8b39cc6130306a51096848559f639"} Jul 16 21:49:55.757383 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:55.756879 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" event={"ID":"6b13a4c5-ceaf-4561-a854-f10942a0369f","Type":"ContainerStarted","Data":"11d3dbb60a67ffb1c9cc4a4c7f123b2c62d4010572391001254b4b933ca7643b"} Jul 16 21:49:55.778605 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:49:55.778549 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" podStartSLOduration=1.582154321 podStartE2EDuration="1.778505474s" totalImagesPullingTime="196.351153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:55.777343413 +0000 UTC m=+642.819567651" watchObservedRunningTime="2026-07-16 21:49:55.778505474 +0000 UTC m=+642.820729750" Jul 16 21:50:04.797222 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:04.797185 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:50:04.797670 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:04.797281 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:50:04.809933 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:04.809895 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:50:05.816426 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:05.816395 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x4lsv" Jul 16 21:50:14.188857 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.188820 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65"] Jul 16 21:50:14.191623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.191597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.194215 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.194187 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 21:50:14.207123 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.207083 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65"] Jul 16 21:50:14.226844 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.226812 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:14.354325 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.354279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59pk\" (UniqueName: \"kubernetes.io/projected/31c41c1f-05a3-40d6-ba9c-e1f077604386-kube-api-access-h59pk\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.354566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.354332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.354566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.354418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.354566 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.354467 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.354757 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.354612 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.354757 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.354680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31c41c1f-05a3-40d6-ba9c-e1f077604386-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.455919 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.455815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59pk\" (UniqueName: \"kubernetes.io/projected/31c41c1f-05a3-40d6-ba9c-e1f077604386-kube-api-access-h59pk\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.455919 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.455878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.455919 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.455911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.456204 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.455961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.456204 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.456073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.456204 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.456155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31c41c1f-05a3-40d6-ba9c-e1f077604386-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.456401 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.456383 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.456487 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.456467 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.456576 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.456471 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.458288 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.458266 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31c41c1f-05a3-40d6-ba9c-e1f077604386-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.458623 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.458604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31c41c1f-05a3-40d6-ba9c-e1f077604386-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.466967 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.466940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59pk\" (UniqueName: \"kubernetes.io/projected/31c41c1f-05a3-40d6-ba9c-e1f077604386-kube-api-access-h59pk\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65\" (UID: \"31c41c1f-05a3-40d6-ba9c-e1f077604386\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.501491 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.501450 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:14.650225 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.650190 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65"] Jul 16 21:50:14.651857 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:50:14.651825 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31c41c1f_05a3_40d6_ba9c_e1f077604386.slice/crio-742cd564751b304326408d956bc78c9cf4393a487a5f9a38d951705c44365718 WatchSource:0}: Error finding container 742cd564751b304326408d956bc78c9cf4393a487a5f9a38d951705c44365718: Status 404 returned error can't find the container with id 742cd564751b304326408d956bc78c9cf4393a487a5f9a38d951705c44365718 Jul 16 21:50:14.857278 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:14.857241 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" event={"ID":"31c41c1f-05a3-40d6-ba9c-e1f077604386","Type":"ContainerStarted","Data":"742cd564751b304326408d956bc78c9cf4393a487a5f9a38d951705c44365718"} Jul 16 21:50:15.863860 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:15.863824 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" event={"ID":"31c41c1f-05a3-40d6-ba9c-e1f077604386","Type":"ContainerStarted","Data":"c86c623a4ebcdcf6a32e865da3a98105563f1f0f19be6c25313001369cfd7e1d"} Jul 16 21:50:15.884223 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:15.884155 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" podStartSLOduration=1.691538419 podStartE2EDuration="1.884136844s" totalImagesPullingTime="192.598425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:15.882207418 +0000 UTC m=+662.924431657" watchObservedRunningTime="2026-07-16 21:50:15.884136844 +0000 UTC m=+662.926361082" Jul 16 21:50:24.501631 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:24.501593 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:24.501631 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:24.501643 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:24.514642 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:24.514612 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:24.920987 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:24.920956 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65" Jul 16 21:50:27.704139 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:27.704093 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:27.767392 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:27.767357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:31.556772 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:31.556730 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:31.623232 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:50:31.623188 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:11.368441 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:11.368313 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:11.461179 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:11.461136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:12.513337 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:12.513302 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:12.600233 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:12.600184 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:13.073270 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.073228 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b9597d5f-9mhgs"] Jul 16 21:51:13.073484 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.073435 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" podUID="1400e9e9-978b-4564-9f4b-67ad29ca29e3" containerName="authorino" containerID="cri-o://bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6" gracePeriod=30 Jul 16 21:51:13.339904 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.339811 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:51:13.442362 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.442323 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxct5\" (UniqueName: \"kubernetes.io/projected/1400e9e9-978b-4564-9f4b-67ad29ca29e3-kube-api-access-rxct5\") pod \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\" (UID: \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\") " Jul 16 21:51:13.442573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.442499 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1400e9e9-978b-4564-9f4b-67ad29ca29e3-tls-cert\") pod \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\" (UID: \"1400e9e9-978b-4564-9f4b-67ad29ca29e3\") " Jul 16 21:51:13.444586 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.444536 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1400e9e9-978b-4564-9f4b-67ad29ca29e3-kube-api-access-rxct5" (OuterVolumeSpecName: "kube-api-access-rxct5") pod "1400e9e9-978b-4564-9f4b-67ad29ca29e3" (UID: "1400e9e9-978b-4564-9f4b-67ad29ca29e3"). InnerVolumeSpecName "kube-api-access-rxct5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:51:13.453643 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.453611 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1400e9e9-978b-4564-9f4b-67ad29ca29e3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1400e9e9-978b-4564-9f4b-67ad29ca29e3" (UID: "1400e9e9-978b-4564-9f4b-67ad29ca29e3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:51:13.543228 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.543194 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxct5\" (UniqueName: \"kubernetes.io/projected/1400e9e9-978b-4564-9f4b-67ad29ca29e3-kube-api-access-rxct5\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:51:13.543228 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:13.543221 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1400e9e9-978b-4564-9f4b-67ad29ca29e3-tls-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:51:14.166851 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.166816 2573 generic.go:356] "Generic (PLEG): container finished" podID="1400e9e9-978b-4564-9f4b-67ad29ca29e3" containerID="bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6" exitCode=0 Jul 16 21:51:14.167045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.166860 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" Jul 16 21:51:14.167045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.166863 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" event={"ID":"1400e9e9-978b-4564-9f4b-67ad29ca29e3","Type":"ContainerDied","Data":"bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6"} Jul 16 21:51:14.167045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.166894 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b9597d5f-9mhgs" event={"ID":"1400e9e9-978b-4564-9f4b-67ad29ca29e3","Type":"ContainerDied","Data":"ca20aca4b7a850387976cb83b612a3497fb2753a16c17713c3cde8054521e0ed"} Jul 16 21:51:14.167045 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.166911 2573 scope.go:117] "RemoveContainer" containerID="bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6" Jul 16 21:51:14.176024 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.176003 2573 scope.go:117] "RemoveContainer" containerID="bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6" Jul 16 21:51:14.176370 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:51:14.176345 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6\": container with ID starting with bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6 not found: ID does not exist" containerID="bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6" Jul 16 21:51:14.176453 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.176379 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6"} err="failed to get container status \"bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6\": rpc error: code = NotFound desc = could not find container \"bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6\": container with ID starting with bad84074cbea15632b9c3d12fcb23ec3e192ce68acfffac4bad2ae3d2d311fd6 not found: ID does not exist" Jul 16 21:51:14.187911 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.187874 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b9597d5f-9mhgs"] Jul 16 21:51:14.195862 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:14.195828 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64b9597d5f-9mhgs"] Jul 16 21:51:15.541007 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:15.540974 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1400e9e9-978b-4564-9f4b-67ad29ca29e3" path="/var/lib/kubelet/pods/1400e9e9-978b-4564-9f4b-67ad29ca29e3/volumes" Jul 16 21:51:32.098921 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:32.098882 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:34.334423 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:51:34.334378 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:22.847577 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:52:22.847538 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:23.968256 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:52:23.968215 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:51.926280 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:52:51.926244 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:52.026300 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:52:52.026254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:06.607123 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:06.607082 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:06.695634 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:06.695598 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:06.795746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:06.795705 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:12.870205 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:12.870171 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:12.941795 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:12.941766 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:13.028393 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:13.028357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:13.126042 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:13.125957 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:13.226217 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:13.226177 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:19.119065 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:19.119029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:19.224861 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:19.224829 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:48.301809 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:53:48.301768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:18.269703 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:18.269621 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw"] Jul 16 21:57:18.270198 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:18.269820 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" podUID="3cb80963-c28c-41c7-b0ed-73bca4d633e8" containerName="manager" containerID="cri-o://d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c" gracePeriod=10 Jul 16 21:57:21.946259 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:21.946229 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:57:21.973573 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:21.973535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:21.994380 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:21.994276 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-468p2\" (UniqueName: \"kubernetes.io/projected/3cb80963-c28c-41c7-b0ed-73bca4d633e8-kube-api-access-468p2\") pod \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\" (UID: \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\") " Jul 16 21:57:21.994380 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:21.994371 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb80963-c28c-41c7-b0ed-73bca4d633e8-extensions-socket-volume\") pod \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\" (UID: \"3cb80963-c28c-41c7-b0ed-73bca4d633e8\") " Jul 16 21:57:21.994796 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:21.994771 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb80963-c28c-41c7-b0ed-73bca4d633e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3cb80963-c28c-41c7-b0ed-73bca4d633e8" (UID: "3cb80963-c28c-41c7-b0ed-73bca4d633e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:57:21.996427 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:21.996393 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb80963-c28c-41c7-b0ed-73bca4d633e8-kube-api-access-468p2" (OuterVolumeSpecName: "kube-api-access-468p2") pod "3cb80963-c28c-41c7-b0ed-73bca4d633e8" (UID: "3cb80963-c28c-41c7-b0ed-73bca4d633e8"). InnerVolumeSpecName "kube-api-access-468p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:57:22.020199 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.020165 2573 generic.go:356] "Generic (PLEG): container finished" podID="3cb80963-c28c-41c7-b0ed-73bca4d633e8" containerID="d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c" exitCode=0 Jul 16 21:57:22.020402 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.020209 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" Jul 16 21:57:22.020402 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.020226 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" event={"ID":"3cb80963-c28c-41c7-b0ed-73bca4d633e8","Type":"ContainerDied","Data":"d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c"} Jul 16 21:57:22.020402 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.020294 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw" event={"ID":"3cb80963-c28c-41c7-b0ed-73bca4d633e8","Type":"ContainerDied","Data":"8d941af126ffc77042811dbf30ff1d3db8c25a88822ddb04857373e44cd564e5"} Jul 16 21:57:22.020402 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.020330 2573 scope.go:117] "RemoveContainer" containerID="d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c" Jul 16 21:57:22.031710 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.031687 2573 scope.go:117] "RemoveContainer" containerID="d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c" Jul 16 21:57:22.032056 ip-10-0-133-93 kubenswrapper[2573]: E0716 21:57:22.032030 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c\": container with ID starting with d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c not found: ID does not exist" containerID="d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c" Jul 16 21:57:22.032173 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.032071 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c"} err="failed to get container status \"d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c\": rpc error: code = NotFound desc = could not find container \"d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c\": container with ID starting with d1510d8d1a2333e5a99858adfc5c35a862b82f636f222cfac8bf715fc652da1c not found: ID does not exist" Jul 16 21:57:22.043402 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.043357 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw"] Jul 16 21:57:22.047657 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.047622 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4nslw"] Jul 16 21:57:22.096159 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.096104 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-468p2\" (UniqueName: \"kubernetes.io/projected/3cb80963-c28c-41c7-b0ed-73bca4d633e8-kube-api-access-468p2\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:57:22.096159 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:22.096156 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb80963-c28c-41c7-b0ed-73bca4d633e8-extensions-socket-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 21:57:23.412138 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:23.412105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:23.540746 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:57:23.540708 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb80963-c28c-41c7-b0ed-73bca4d633e8" path="/var/lib/kubelet/pods/3cb80963-c28c-41c7-b0ed-73bca4d633e8/volumes" Jul 16 21:58:19.375033 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.374997 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb"] Jul 16 21:58:19.375510 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.375399 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb80963-c28c-41c7-b0ed-73bca4d633e8" containerName="manager" Jul 16 21:58:19.375510 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.375410 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb80963-c28c-41c7-b0ed-73bca4d633e8" containerName="manager" Jul 16 21:58:19.375510 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.375419 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1400e9e9-978b-4564-9f4b-67ad29ca29e3" containerName="authorino" Jul 16 21:58:19.375510 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.375426 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1400e9e9-978b-4564-9f4b-67ad29ca29e3" containerName="authorino" Jul 16 21:58:19.375510 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.375482 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1400e9e9-978b-4564-9f4b-67ad29ca29e3" containerName="authorino" Jul 16 21:58:19.375510 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.375495 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb80963-c28c-41c7-b0ed-73bca4d633e8" containerName="manager" Jul 16 21:58:19.379641 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.379622 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:19.382971 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.382947 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 21:58:19.383552 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.383533 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 21:58:19.383712 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.383694 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mvd4n\"" Jul 16 21:58:19.396378 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.396342 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb"] Jul 16 21:58:19.545941 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.545906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hffw8\" (UniqueName: \"kubernetes.io/projected/68af045e-6189-4916-97f2-9959ad046053-kube-api-access-hffw8\") pod \"kuadrant-operator-controller-manager-86878c769f-mzrbb\" (UID: \"68af045e-6189-4916-97f2-9959ad046053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:19.546113 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.545967 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68af045e-6189-4916-97f2-9959ad046053-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzrbb\" (UID: \"68af045e-6189-4916-97f2-9959ad046053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:19.647307 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.647216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hffw8\" (UniqueName: \"kubernetes.io/projected/68af045e-6189-4916-97f2-9959ad046053-kube-api-access-hffw8\") pod \"kuadrant-operator-controller-manager-86878c769f-mzrbb\" (UID: \"68af045e-6189-4916-97f2-9959ad046053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:19.647480 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.647317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68af045e-6189-4916-97f2-9959ad046053-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzrbb\" (UID: \"68af045e-6189-4916-97f2-9959ad046053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:19.647823 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.647799 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68af045e-6189-4916-97f2-9959ad046053-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzrbb\" (UID: \"68af045e-6189-4916-97f2-9959ad046053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:19.657473 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.657443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hffw8\" (UniqueName: \"kubernetes.io/projected/68af045e-6189-4916-97f2-9959ad046053-kube-api-access-hffw8\") pod \"kuadrant-operator-controller-manager-86878c769f-mzrbb\" (UID: \"68af045e-6189-4916-97f2-9959ad046053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:19.697080 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.697043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:19.847338 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:19.847308 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb"] Jul 16 21:58:19.849682 ip-10-0-133-93 kubenswrapper[2573]: W0716 21:58:19.849654 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68af045e_6189_4916_97f2_9959ad046053.slice/crio-4c513fd2e0b47d00f0d09669bbf3a4dffc08618e256d5b0069fd87a469a81b5a WatchSource:0}: Error finding container 4c513fd2e0b47d00f0d09669bbf3a4dffc08618e256d5b0069fd87a469a81b5a: Status 404 returned error can't find the container with id 4c513fd2e0b47d00f0d09669bbf3a4dffc08618e256d5b0069fd87a469a81b5a Jul 16 21:58:20.315940 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:20.315902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" event={"ID":"68af045e-6189-4916-97f2-9959ad046053","Type":"ContainerStarted","Data":"cd9879cf148cfe58572f8b4b5ad49df5be5366263b28bd19bc348a7dbe38e325"} Jul 16 21:58:20.316153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:20.315954 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" event={"ID":"68af045e-6189-4916-97f2-9959ad046053","Type":"ContainerStarted","Data":"4c513fd2e0b47d00f0d09669bbf3a4dffc08618e256d5b0069fd87a469a81b5a"} Jul 16 21:58:20.316153 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:20.316033 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:20.340738 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:20.340681 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" podStartSLOduration=1.340661113 podStartE2EDuration="1.340661113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:20.339068742 +0000 UTC m=+1147.381292981" watchObservedRunningTime="2026-07-16 21:58:20.340661113 +0000 UTC m=+1147.382885409" Jul 16 21:58:27.553477 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:27.553441 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:27.654185 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:27.654151 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:31.323605 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:31.323574 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzrbb" Jul 16 21:58:50.065867 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:50.065827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:55.332284 ip-10-0-133-93 kubenswrapper[2573]: I0716 21:58:55.332243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:00:55.595772 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:00:55.595727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:22.811669 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:22.811620 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:28.870875 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:28.870832 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:37.118399 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:37.118319 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:42.212754 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:42.212713 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:44.949707 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:44.949670 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:45.037343 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:45.037307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:45.366615 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:45.366580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:51.553872 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:51.553840 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:51.618554 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:51.618495 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:52.621305 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:52.621266 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:52.914295 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:52.914202 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:53.015047 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:53.009559 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:53.134655 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:53.134614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:53.362272 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:53.362238 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:53.456545 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:53.456490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:55.213731 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:55.213700 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:57.590627 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:57.590595 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:57.898308 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:57.898219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:57.993844 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:57.993805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:58.161282 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:58.161196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:04:59.491192 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:04:59.491160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:04.028674 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:04.028633 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:04.144206 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:04.144175 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:04.292291 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:04.292201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:04.544563 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:04.544444 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:05.465937 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:05.465900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:05.550175 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:05.550145 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:05.655347 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:05.655296 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:06.103749 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:06.103710 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:06.203396 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:06.203363 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:06.304885 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:06.304848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:07.727952 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:07.727907 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:10.451538 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:10.451490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:10.835675 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:10.835645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:16.504357 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:16.504320 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:16.916308 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:16.916269 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:18.654787 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:18.654755 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:22.378202 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:22.378167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:22.489904 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:22.489866 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:22.759697 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:22.759614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:22.858678 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:22.858646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:24.231531 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:24.231496 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:24.360322 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:24.360285 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:24.632919 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:24.632879 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:24.735911 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:24.735845 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:26.390377 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:26.390333 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:28.535934 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:28.535900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:28.763502 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:28.763464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:30.614107 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:30.614071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:34.353765 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:34.353722 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:34.419464 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:34.419427 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:34.861022 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:34.860986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:34.950757 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:34.950722 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:35.053742 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:35.053711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:36.394483 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:36.394444 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:42.224807 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:42.224770 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:42.511290 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:42.511180 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:42.612439 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:42.612398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:42.930778 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:05:42.930749 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:08.893032 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:08.892939 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:14.984570 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:14.984511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:15.106869 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:15.106831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:15.214716 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:15.214673 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:25.632646 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:25.632581 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:31.049185 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:31.049148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:36.820596 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:36.820548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:36.929168 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:36.929128 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:37.020089 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:37.020055 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:43.172273 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:43.172234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:43.234503 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:43.234474 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:47.575372 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:47.575321 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:48.346483 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:48.346433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:54.239601 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:54.239571 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:54.349298 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:54.349244 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:54.441201 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:54.441169 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:54.539913 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:54.539821 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:55.301433 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:55.301398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:55.401130 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:55.401087 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:55.497568 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:55.497507 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:06:57.112697 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:06:57.112659 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:02.909496 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:02.909460 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:02.970040 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:02.970009 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:03.567726 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:03.567690 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:09.275202 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:09.275165 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:09.375091 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:09.375056 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:09.703945 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:09.703906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:10.039380 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:10.039298 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:10.136758 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:10.136722 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:12.933316 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:12.933283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:13.021262 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:13.021230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:13.117752 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:13.117713 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:13.230418 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:13.230332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:13.603783 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:13.603737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:13.705206 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:13.705163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:17.416373 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:17.416336 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:18.727776 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:18.727727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:18.885159 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:18.885117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:19.320317 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:19.320270 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:25.129414 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:25.129364 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:25.197042 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:25.197006 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:25.568568 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:25.568541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:26.308407 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:26.308370 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:26.364300 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:26.364269 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:32.458021 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:32.457984 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:32.530126 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:32.530068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:32.675118 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:32.675087 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:32.802067 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:32.802035 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:32.896236 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:32.896196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:33.005128 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:33.005096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:38.833741 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:38.833700 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:38.945133 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:38.945091 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:39.036486 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:39.036451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:39.189757 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:39.189676 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:44.924674 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:44.924639 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:44.996837 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:44.996802 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:48.450191 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:48.450157 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:48.554688 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:48.554656 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:52.335267 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:52.335231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:52.637090 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:52.636998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:53.063095 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:53.063062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:55.096735 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:55.096679 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:55.196345 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:55.196300 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:55.505247 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:55.505127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:55.607290 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:55.607254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:55.882200 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:55.882158 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:55.983083 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:55.983039 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:07:59.581138 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:07:59.581098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:08:08.715846 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:08:08.715810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:08:26.777956 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:08:26.777925 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:08:27.141734 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:08:27.141687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:08:32.850360 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:08:32.850310 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:08:37.925627 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:08:37.925583 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:08:44.472197 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:08:44.472162 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:14.452680 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:14.452629 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:14.944945 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:14.944891 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:20.447915 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:20.447874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:20.554229 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:20.554199 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:20.733008 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:20.732917 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:20.967388 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:20.967357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:22.012431 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:22.012394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:22.117603 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:22.117572 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:22.217299 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:22.217266 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:22.893424 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:22.893388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:22.993745 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:22.993705 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:23.104182 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:23.104149 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:24.471790 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:24.471760 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:30.459383 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:30.459345 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:32.082426 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:32.082391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:36.824193 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:36.824160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:36.934469 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:36.934430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:37.028223 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:37.028181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:47.531086 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:47.531041 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:09:47.869663 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:09:47.869629 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:08.911055 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:08.911022 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:14.917255 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:14.917216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:14.994986 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:14.994944 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:15.280356 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:15.280278 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:15.664864 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:15.664831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:15.828842 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:15.828802 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:18.984701 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:18.984666 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:19.104041 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:19.104010 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:19.214442 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:19.214411 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:19.667012 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:19.666970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:19.766127 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:19.766078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:19.868270 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:19.868237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:23.443767 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:23.443725 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:23.775231 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:23.775149 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:24.190103 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:24.190067 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:24.683861 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:24.683822 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:27.019163 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:27.019129 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:27.138304 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:27.138255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:27.248213 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:27.248183 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:27.517266 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:27.517233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:28.056262 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:28.056232 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:28.178261 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:28.178229 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:31.504296 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:31.504265 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:40.546879 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:40.546844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:40.650024 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:40.649988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:47.150188 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.150143 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8"] Jul 16 22:10:47.153968 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.153943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.157736 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.157711 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9699fb\"/\"kube-root-ca.crt\"" Jul 16 22:10:47.159664 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.159642 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9699fb\"/\"openshift-service-ca.crt\"" Jul 16 22:10:47.159804 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.159764 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9699fb\"/\"default-dockercfg-f6lbs\"" Jul 16 22:10:47.160723 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.160702 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9699fb\"/\"test-model-9699fb-kserve-self-signed-certs\"" Jul 16 22:10:47.191511 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.191477 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8"] Jul 16 22:10:47.228337 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.228303 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:47.332408 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.332369 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-tmp-dir\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.332408 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.332411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-home\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.332677 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.332433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-model-cache\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.332677 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.332486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-dshm\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.332677 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.332626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhx9\" (UniqueName: \"kubernetes.io/projected/a82d180a-48ad-43e3-8efb-498376d8586b-kube-api-access-gbhx9\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.332677 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.332670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82d180a-48ad-43e3-8efb-498376d8586b-tls-certs\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.433467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-tmp-dir\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.433467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-home\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.433467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-model-cache\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.433467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-dshm\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.433852 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433499 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhx9\" (UniqueName: \"kubernetes.io/projected/a82d180a-48ad-43e3-8efb-498376d8586b-kube-api-access-gbhx9\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.433852 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82d180a-48ad-43e3-8efb-498376d8586b-tls-certs\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.433966 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-tmp-dir\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.433966 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433909 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-home\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.434049 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.433962 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-model-cache\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.435900 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.435873 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-dshm\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.436065 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.436047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82d180a-48ad-43e3-8efb-498376d8586b-tls-certs\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.443348 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.443321 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhx9\" (UniqueName: \"kubernetes.io/projected/a82d180a-48ad-43e3-8efb-498376d8586b-kube-api-access-gbhx9\") pod \"test-model-9699fb-kserve-b55c594f9-ptln8\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.464094 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.464057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:47.617512 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:47.617483 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8"] Jul 16 22:10:47.618788 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:10:47.618756 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda82d180a_48ad_43e3_8efb_498376d8586b.slice/crio-aa93de83888b3be82721d562398b099d751ce3a2c9515c56eeb4c3a7c96df365 WatchSource:0}: Error finding container aa93de83888b3be82721d562398b099d751ce3a2c9515c56eeb4c3a7c96df365: Status 404 returned error can't find the container with id aa93de83888b3be82721d562398b099d751ce3a2c9515c56eeb4c3a7c96df365 Jul 16 22:10:48.121017 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:48.120979 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" event={"ID":"a82d180a-48ad-43e3-8efb-498376d8586b","Type":"ContainerStarted","Data":"06fafcec81423ae5abb9a113b593b326e379483ea98fada242d405920df8e46a"} Jul 16 22:10:48.121199 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:48.121028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" event={"ID":"a82d180a-48ad-43e3-8efb-498376d8586b","Type":"ContainerStarted","Data":"aa93de83888b3be82721d562398b099d751ce3a2c9515c56eeb4c3a7c96df365"} Jul 16 22:10:48.152156 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:48.152100 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" podStartSLOduration=1.152082925 podStartE2EDuration="1.152082925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:10:48.148994634 +0000 UTC m=+1895.191218872" watchObservedRunningTime="2026-07-16 22:10:48.152082925 +0000 UTC m=+1895.194307162" Jul 16 22:10:48.921264 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:48.921231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:49.023099 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:49.023048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:57.464781 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:57.464732 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:57.464781 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:57.464791 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:57.477456 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:57.477425 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:10:58.180103 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:10:58.180068 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:11:24.227822 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:24.227789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:45.332262 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:45.332220 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8"] Jul 16 22:11:45.357569 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:45.357514 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:46.266513 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:46.266477 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:46.417060 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:46.417029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:46.808429 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:46.808394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:46.909126 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:46.909093 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:47.010637 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:47.010599 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:47.138393 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:47.138307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:51.879479 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:11:51.879433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:00.352891 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.352845 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" podUID="a82d180a-48ad-43e3-8efb-498376d8586b" containerName="main" containerID="cri-o://06fafcec81423ae5abb9a113b593b326e379483ea98fada242d405920df8e46a" gracePeriod=45 Jul 16 22:12:00.478277 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.478246 2573 generic.go:356] "Generic (PLEG): container finished" podID="a82d180a-48ad-43e3-8efb-498376d8586b" containerID="06fafcec81423ae5abb9a113b593b326e379483ea98fada242d405920df8e46a" exitCode=0 Jul 16 22:12:00.478426 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.478323 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" event={"ID":"a82d180a-48ad-43e3-8efb-498376d8586b","Type":"ContainerDied","Data":"06fafcec81423ae5abb9a113b593b326e379483ea98fada242d405920df8e46a"} Jul 16 22:12:00.613821 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.613739 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:12:00.762800 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.762766 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbhx9\" (UniqueName: \"kubernetes.io/projected/a82d180a-48ad-43e3-8efb-498376d8586b-kube-api-access-gbhx9\") pod \"a82d180a-48ad-43e3-8efb-498376d8586b\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " Jul 16 22:12:00.762986 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.762846 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-tmp-dir\") pod \"a82d180a-48ad-43e3-8efb-498376d8586b\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " Jul 16 22:12:00.762986 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.762922 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-model-cache\") pod \"a82d180a-48ad-43e3-8efb-498376d8586b\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " Jul 16 22:12:00.762986 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.762951 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-dshm\") pod \"a82d180a-48ad-43e3-8efb-498376d8586b\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " Jul 16 22:12:00.762986 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.762971 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-home\") pod \"a82d180a-48ad-43e3-8efb-498376d8586b\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " Jul 16 22:12:00.763153 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.763005 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82d180a-48ad-43e3-8efb-498376d8586b-tls-certs\") pod \"a82d180a-48ad-43e3-8efb-498376d8586b\" (UID: \"a82d180a-48ad-43e3-8efb-498376d8586b\") " Jul 16 22:12:00.763153 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.763097 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a82d180a-48ad-43e3-8efb-498376d8586b" (UID: "a82d180a-48ad-43e3-8efb-498376d8586b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:12:00.763295 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.763258 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a82d180a-48ad-43e3-8efb-498376d8586b" (UID: "a82d180a-48ad-43e3-8efb-498376d8586b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:12:00.763295 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.763272 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-home" (OuterVolumeSpecName: "home") pod "a82d180a-48ad-43e3-8efb-498376d8586b" (UID: "a82d180a-48ad-43e3-8efb-498376d8586b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:12:00.763464 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.763327 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:12:00.765134 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.765107 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-dshm" (OuterVolumeSpecName: "dshm") pod "a82d180a-48ad-43e3-8efb-498376d8586b" (UID: "a82d180a-48ad-43e3-8efb-498376d8586b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:12:00.765234 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.765154 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a82d180a-48ad-43e3-8efb-498376d8586b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a82d180a-48ad-43e3-8efb-498376d8586b" (UID: "a82d180a-48ad-43e3-8efb-498376d8586b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:12:00.765234 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.765206 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82d180a-48ad-43e3-8efb-498376d8586b-kube-api-access-gbhx9" (OuterVolumeSpecName: "kube-api-access-gbhx9") pod "a82d180a-48ad-43e3-8efb-498376d8586b" (UID: "a82d180a-48ad-43e3-8efb-498376d8586b"). InnerVolumeSpecName "kube-api-access-gbhx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:12:00.864456 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.864351 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:12:00.864456 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.864384 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:12:00.864456 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.864397 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82d180a-48ad-43e3-8efb-498376d8586b-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:12:00.864456 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.864409 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82d180a-48ad-43e3-8efb-498376d8586b-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:12:00.864456 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:00.864422 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbhx9\" (UniqueName: \"kubernetes.io/projected/a82d180a-48ad-43e3-8efb-498376d8586b-kube-api-access-gbhx9\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:12:01.485310 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:01.485265 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" event={"ID":"a82d180a-48ad-43e3-8efb-498376d8586b","Type":"ContainerDied","Data":"aa93de83888b3be82721d562398b099d751ce3a2c9515c56eeb4c3a7c96df365"} Jul 16 22:12:01.485744 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:01.485289 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8" Jul 16 22:12:01.485744 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:01.485323 2573 scope.go:117] "RemoveContainer" containerID="06fafcec81423ae5abb9a113b593b326e379483ea98fada242d405920df8e46a" Jul 16 22:12:01.507939 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:01.507908 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8"] Jul 16 22:12:01.511642 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:01.511615 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9699fb/test-model-9699fb-kserve-b55c594f9-ptln8"] Jul 16 22:12:01.542451 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:01.542418 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82d180a-48ad-43e3-8efb-498376d8586b" path="/var/lib/kubelet/pods/a82d180a-48ad-43e3-8efb-498376d8586b/volumes" Jul 16 22:12:28.751095 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:28.751063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:29.101164 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:29.101129 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:29.337899 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:29.337863 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:29.481239 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:29.481150 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:29.870222 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:29.870189 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:30.206093 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:30.206003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:30.304457 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:30.304427 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:30.502850 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:30.502758 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:36.001107 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:36.001072 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:36.099923 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:36.099888 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:36.268392 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:36.268307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:36.508856 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:36.508822 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:36.857058 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:36.857027 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:37.190338 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:37.190231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:37.470810 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:37.470723 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:39.765762 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:39.765726 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:39.879457 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:39.879426 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:39.981699 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:39.981668 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:40.082982 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:40.082953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:40.526328 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:40.526243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:40.626480 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:40.626445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:44.176037 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:44.176000 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:44.640240 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:44.640206 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:44.955290 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:44.955205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:45.335148 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:45.335113 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:47.728402 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:47.728369 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:47.839749 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:47.839715 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:47.927090 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:47.927051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:48.477264 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:48.477230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:48.576826 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:48.576790 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:48.677085 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:48.677057 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:49.252114 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:49.252077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:49.352677 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:49.352642 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:50.111122 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:50.111077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:50.211651 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:50.211615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:50.653289 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:50.653262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:50.713040 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:50.713012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:52.241428 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:52.241387 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:56.873502 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:12:56.873467 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:00.305038 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:00.305003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:00.692990 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:00.692854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:00.793244 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:00.793190 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:00.893836 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:00.893804 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:01.038274 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:01.038170 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:06.551722 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:06.551687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:06.652332 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:06.652296 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:06.805550 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:06.805440 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:07.054028 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:07.053989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:07.349076 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:07.348996 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:07.619261 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:07.619154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:07.721236 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:07.721201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:07.819967 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:07.819921 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:07.923294 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:07.923197 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:08.108218 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:08.108187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:13.427260 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:13.427220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:13.489092 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:13.489059 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:13.675802 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:13.675769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:13.891132 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:13.891092 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:14.217213 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:14.217122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:14.525327 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:14.525239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:14.799728 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:14.799696 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:17.919582 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:17.919511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:18.018332 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:18.018299 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:18.122142 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:18.122111 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:18.233800 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:18.233709 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:18.570254 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:18.570224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:18.673374 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:18.673323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:22.369613 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:22.369577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:22.702755 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:22.702637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:22.994857 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:22.994740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:23.406955 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:23.406919 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:25.945229 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:25.945196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:26.028975 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:26.028937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:26.105683 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:26.105622 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:26.489337 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:26.489302 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:26.649216 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:26.649181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:26.757018 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:26.756935 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:30.456688 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:30.456650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:39.316427 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:39.316388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:39.416266 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:39.416227 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:40.112794 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:40.112742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:40.164312 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:40.164276 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:48.450578 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:48.450546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:48.551346 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:48.551314 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:57.699955 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.699914 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8"] Jul 16 22:13:57.700464 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.700337 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82d180a-48ad-43e3-8efb-498376d8586b" containerName="main" Jul 16 22:13:57.700464 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.700347 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82d180a-48ad-43e3-8efb-498376d8586b" containerName="main" Jul 16 22:13:57.700464 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.700411 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82d180a-48ad-43e3-8efb-498376d8586b" containerName="main" Jul 16 22:13:57.703583 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.703558 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.706489 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.706457 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d61a3\"/\"openshift-service-ca.crt\"" Jul 16 22:13:57.706665 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.706578 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d61a3\"/\"auth-test-model-5d61a3-kserve-self-signed-certs\"" Jul 16 22:13:57.707365 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.707340 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d61a3\"/\"kube-root-ca.crt\"" Jul 16 22:13:57.707482 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.707403 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d61a3\"/\"default-dockercfg-6w9vs\"" Jul 16 22:13:57.713798 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.713764 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8"] Jul 16 22:13:57.739424 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.739380 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:13:57.818994 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.818947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665rn\" (UniqueName: \"kubernetes.io/projected/1a8ab3ab-4247-48e7-bfea-89e13524a951-kube-api-access-665rn\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.819180 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.819040 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-tmp-dir\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.819180 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.819065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8ab3ab-4247-48e7-bfea-89e13524a951-tls-certs\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.819180 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.819084 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-model-cache\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.819290 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.819194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-home\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.819290 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.819221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-dshm\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.920790 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.920751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-tmp-dir\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.920790 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.920792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8ab3ab-4247-48e7-bfea-89e13524a951-tls-certs\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.921126 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.920815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-model-cache\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.921126 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.920861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-home\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.921126 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.920879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-dshm\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.921126 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.920921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665rn\" (UniqueName: \"kubernetes.io/projected/1a8ab3ab-4247-48e7-bfea-89e13524a951-kube-api-access-665rn\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.921338 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.921274 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-model-cache\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.921338 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.921300 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-tmp-dir\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.921408 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.921351 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-home\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.923193 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.923161 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-dshm\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.923373 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.923355 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8ab3ab-4247-48e7-bfea-89e13524a951-tls-certs\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:57.930339 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:57.930310 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-665rn\" (UniqueName: \"kubernetes.io/projected/1a8ab3ab-4247-48e7-bfea-89e13524a951-kube-api-access-665rn\") pod \"auth-test-model-5d61a3-kserve-597595d79f-z8kj8\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:58.014408 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:58.014310 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:13:58.158110 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:58.158082 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8"] Jul 16 22:13:58.160232 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:13:58.160205 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a8ab3ab_4247_48e7_bfea_89e13524a951.slice/crio-66b9435fad3fb814eb39a49f3e98590a06dc1dc84444358cafc385f5b7140975 WatchSource:0}: Error finding container 66b9435fad3fb814eb39a49f3e98590a06dc1dc84444358cafc385f5b7140975: Status 404 returned error can't find the container with id 66b9435fad3fb814eb39a49f3e98590a06dc1dc84444358cafc385f5b7140975 Jul 16 22:13:59.090677 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:59.090641 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" event={"ID":"1a8ab3ab-4247-48e7-bfea-89e13524a951","Type":"ContainerStarted","Data":"cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53"} Jul 16 22:13:59.090677 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:59.090686 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" event={"ID":"1a8ab3ab-4247-48e7-bfea-89e13524a951","Type":"ContainerStarted","Data":"66b9435fad3fb814eb39a49f3e98590a06dc1dc84444358cafc385f5b7140975"} Jul 16 22:13:59.112258 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:13:59.112197 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" podStartSLOduration=2.11218021 podStartE2EDuration="2.11218021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:13:59.109445691 +0000 UTC m=+2086.151669928" watchObservedRunningTime="2026-07-16 22:13:59.11218021 +0000 UTC m=+2086.154404447" Jul 16 22:14:05.414103 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:14:05.414068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:08.015281 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:14:08.015232 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:14:08.015733 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:14:08.015305 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:14:08.027792 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:14:08.027762 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:14:08.148989 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:14:08.148951 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:14:28.429860 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:14:28.429798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:16:29.221759 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:29.221726 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8"] Jul 16 22:16:29.278883 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:29.278850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:16:30.770622 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:30.770580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:16:31.916412 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:31.916373 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:16:37.193060 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.192967 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt"] Jul 16 22:16:37.196712 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.196689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.199202 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.199177 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d61a3\"/\"sub-test-model-5d61a3-kserve-self-signed-certs\"" Jul 16 22:16:37.212913 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.212879 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt"] Jul 16 22:16:37.231094 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.231061 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:16:37.313937 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.313898 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-model-cache\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.314120 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.313956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxbwx\" (UniqueName: \"kubernetes.io/projected/48278cd8-746e-470d-a734-5fe3c0c40f0a-kube-api-access-mxbwx\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.314120 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.314016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48278cd8-746e-470d-a734-5fe3c0c40f0a-tls-certs\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.314120 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.314036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-home\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.314120 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.314062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-dshm\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.314120 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.314094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-tmp-dir\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415005 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.414959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxbwx\" (UniqueName: \"kubernetes.io/projected/48278cd8-746e-470d-a734-5fe3c0c40f0a-kube-api-access-mxbwx\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415212 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.415024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48278cd8-746e-470d-a734-5fe3c0c40f0a-tls-certs\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415212 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.415044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-home\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415212 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.415067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-dshm\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415212 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.415102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-tmp-dir\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415212 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.415174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-model-cache\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415502 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.415480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-home\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415622 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.415597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-model-cache\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.415665 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.415623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-tmp-dir\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.417398 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.417376 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-dshm\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.417811 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.417786 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48278cd8-746e-470d-a734-5fe3c0c40f0a-tls-certs\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.424601 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.424573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxbwx\" (UniqueName: \"kubernetes.io/projected/48278cd8-746e-470d-a734-5fe3c0c40f0a-kube-api-access-mxbwx\") pod \"sub-test-model-5d61a3-kserve-8df7588c9-nxkzt\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.510118 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.510021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:37.650299 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.650267 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt"] Jul 16 22:16:37.652266 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:16:37.652231 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48278cd8_746e_470d_a734_5fe3c0c40f0a.slice/crio-a3ff192e1cea4188e17cbffdbd3cf12fc380227a8152e0bce7ecca40bb47f8b9 WatchSource:0}: Error finding container a3ff192e1cea4188e17cbffdbd3cf12fc380227a8152e0bce7ecca40bb47f8b9: Status 404 returned error can't find the container with id a3ff192e1cea4188e17cbffdbd3cf12fc380227a8152e0bce7ecca40bb47f8b9 Jul 16 22:16:37.917831 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.917797 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" event={"ID":"48278cd8-746e-470d-a734-5fe3c0c40f0a","Type":"ContainerStarted","Data":"340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba"} Jul 16 22:16:37.917831 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.917839 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" event={"ID":"48278cd8-746e-470d-a734-5fe3c0c40f0a","Type":"ContainerStarted","Data":"a3ff192e1cea4188e17cbffdbd3cf12fc380227a8152e0bce7ecca40bb47f8b9"} Jul 16 22:16:37.937457 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:37.937392 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" podStartSLOduration=0.937370431 podStartE2EDuration="937.370431ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:16:37.935479512 +0000 UTC m=+2244.977703752" watchObservedRunningTime="2026-07-16 22:16:37.937370431 +0000 UTC m=+2244.979594668" Jul 16 22:16:44.244452 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.244405 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" podUID="1a8ab3ab-4247-48e7-bfea-89e13524a951" containerName="main" containerID="cri-o://cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53" gracePeriod=45 Jul 16 22:16:44.496533 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.496449 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:16:44.589182 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589143 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8ab3ab-4247-48e7-bfea-89e13524a951-tls-certs\") pod \"1a8ab3ab-4247-48e7-bfea-89e13524a951\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " Jul 16 22:16:44.589372 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589263 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-665rn\" (UniqueName: \"kubernetes.io/projected/1a8ab3ab-4247-48e7-bfea-89e13524a951-kube-api-access-665rn\") pod \"1a8ab3ab-4247-48e7-bfea-89e13524a951\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " Jul 16 22:16:44.589372 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589285 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-model-cache\") pod \"1a8ab3ab-4247-48e7-bfea-89e13524a951\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " Jul 16 22:16:44.589372 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589305 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-tmp-dir\") pod \"1a8ab3ab-4247-48e7-bfea-89e13524a951\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " Jul 16 22:16:44.589372 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589339 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-home\") pod \"1a8ab3ab-4247-48e7-bfea-89e13524a951\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " Jul 16 22:16:44.589372 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589359 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-dshm\") pod \"1a8ab3ab-4247-48e7-bfea-89e13524a951\" (UID: \"1a8ab3ab-4247-48e7-bfea-89e13524a951\") " Jul 16 22:16:44.589676 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589625 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a8ab3ab-4247-48e7-bfea-89e13524a951" (UID: "1a8ab3ab-4247-48e7-bfea-89e13524a951"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:16:44.589676 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589635 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a8ab3ab-4247-48e7-bfea-89e13524a951" (UID: "1a8ab3ab-4247-48e7-bfea-89e13524a951"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:16:44.589771 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589675 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-home" (OuterVolumeSpecName: "home") pod "1a8ab3ab-4247-48e7-bfea-89e13524a951" (UID: "1a8ab3ab-4247-48e7-bfea-89e13524a951"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:16:44.589831 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589806 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:16:44.589831 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589831 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:16:44.590001 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.589843 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:16:44.591604 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.591577 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-dshm" (OuterVolumeSpecName: "dshm") pod "1a8ab3ab-4247-48e7-bfea-89e13524a951" (UID: "1a8ab3ab-4247-48e7-bfea-89e13524a951"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:16:44.591698 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.591642 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8ab3ab-4247-48e7-bfea-89e13524a951-kube-api-access-665rn" (OuterVolumeSpecName: "kube-api-access-665rn") pod "1a8ab3ab-4247-48e7-bfea-89e13524a951" (UID: "1a8ab3ab-4247-48e7-bfea-89e13524a951"). InnerVolumeSpecName "kube-api-access-665rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:16:44.591740 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.591714 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8ab3ab-4247-48e7-bfea-89e13524a951-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a8ab3ab-4247-48e7-bfea-89e13524a951" (UID: "1a8ab3ab-4247-48e7-bfea-89e13524a951"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:16:44.691397 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.691355 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8ab3ab-4247-48e7-bfea-89e13524a951-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:16:44.691397 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.691390 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665rn\" (UniqueName: \"kubernetes.io/projected/1a8ab3ab-4247-48e7-bfea-89e13524a951-kube-api-access-665rn\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:16:44.691397 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.691406 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8ab3ab-4247-48e7-bfea-89e13524a951-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:16:44.956300 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.956259 2573 generic.go:356] "Generic (PLEG): container finished" podID="1a8ab3ab-4247-48e7-bfea-89e13524a951" containerID="cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53" exitCode=0 Jul 16 22:16:44.956467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.956320 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" Jul 16 22:16:44.956467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.956344 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" event={"ID":"1a8ab3ab-4247-48e7-bfea-89e13524a951","Type":"ContainerDied","Data":"cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53"} Jul 16 22:16:44.956467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.956392 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8" event={"ID":"1a8ab3ab-4247-48e7-bfea-89e13524a951","Type":"ContainerDied","Data":"66b9435fad3fb814eb39a49f3e98590a06dc1dc84444358cafc385f5b7140975"} Jul 16 22:16:44.956467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.956415 2573 scope.go:117] "RemoveContainer" containerID="cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53" Jul 16 22:16:44.965723 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.965700 2573 scope.go:117] "RemoveContainer" containerID="cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53" Jul 16 22:16:44.966030 ip-10-0-133-93 kubenswrapper[2573]: E0716 22:16:44.966009 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53\": container with ID starting with cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53 not found: ID does not exist" containerID="cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53" Jul 16 22:16:44.966086 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.966038 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53"} err="failed to get container status \"cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53\": rpc error: code = NotFound desc = could not find container \"cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53\": container with ID starting with cf5d1d2a87e9963e46e9c624e7071ccd9746390dcdfdb6c9a8641d3eb6134c53 not found: ID does not exist" Jul 16 22:16:44.982218 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.982144 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8"] Jul 16 22:16:44.985614 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:44.985583 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/auth-test-model-5d61a3-kserve-597595d79f-z8kj8"] Jul 16 22:16:45.543979 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:45.543938 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8ab3ab-4247-48e7-bfea-89e13524a951" path="/var/lib/kubelet/pods/1a8ab3ab-4247-48e7-bfea-89e13524a951/volumes" Jul 16 22:16:47.511008 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:47.510971 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:47.511396 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:47.511031 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:47.523824 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:47.523791 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:16:47.983565 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:16:47.983507 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:17:17.591669 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.591631 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5"] Jul 16 22:17:17.592116 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.592035 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8ab3ab-4247-48e7-bfea-89e13524a951" containerName="main" Jul 16 22:17:17.592116 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.592046 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8ab3ab-4247-48e7-bfea-89e13524a951" containerName="main" Jul 16 22:17:17.592116 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.592117 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8ab3ab-4247-48e7-bfea-89e13524a951" containerName="main" Jul 16 22:17:17.595227 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.595203 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.597910 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.597889 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff5f80\"/\"openshift-service-ca.crt\"" Jul 16 22:17:17.598050 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.597891 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff5f80\"/\"kube-root-ca.crt\"" Jul 16 22:17:17.598996 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.598971 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff5f80\"/\"sub-test-model-ff5f80-kserve-self-signed-certs\"" Jul 16 22:17:17.599131 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.598972 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff5f80\"/\"default-dockercfg-nk4tz\"" Jul 16 22:17:17.605813 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.605786 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5"] Jul 16 22:17:17.712223 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.712157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-tmp-dir\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.712223 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.712222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-dshm\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.712468 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.712295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwbh\" (UniqueName: \"kubernetes.io/projected/df5954e5-8adb-4e96-9b36-b19c37018858-kube-api-access-skwbh\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.712468 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.712342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-home\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.712468 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.712368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5954e5-8adb-4e96-9b36-b19c37018858-tls-certs\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.712468 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.712405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-model-cache\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.774730 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.774684 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:17.813987 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.813943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-dshm\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.813987 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.813996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwbh\" (UniqueName: \"kubernetes.io/projected/df5954e5-8adb-4e96-9b36-b19c37018858-kube-api-access-skwbh\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.814264 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.814031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-home\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.814264 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.814057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5954e5-8adb-4e96-9b36-b19c37018858-tls-certs\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.814264 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.814082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-model-cache\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.814264 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.814158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-tmp-dir\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.814488 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.814468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-home\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.814555 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.814539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-tmp-dir\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.814602 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.814578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-model-cache\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.816359 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.816342 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-dshm\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.816475 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.816458 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5954e5-8adb-4e96-9b36-b19c37018858-tls-certs\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.822642 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.822617 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwbh\" (UniqueName: \"kubernetes.io/projected/df5954e5-8adb-4e96-9b36-b19c37018858-kube-api-access-skwbh\") pod \"sub-test-model-ff5f80-kserve-8467d7f997-bc5v5\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:17.905798 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:17.905701 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:18.045976 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:18.045939 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5"] Jul 16 22:17:18.047115 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:17:18.047093 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf5954e5_8adb_4e96_9b36_b19c37018858.slice/crio-2864a74388d9188a72ca45443eedc72e658f66286178eaa251f05aae1cf73b06 WatchSource:0}: Error finding container 2864a74388d9188a72ca45443eedc72e658f66286178eaa251f05aae1cf73b06: Status 404 returned error can't find the container with id 2864a74388d9188a72ca45443eedc72e658f66286178eaa251f05aae1cf73b06 Jul 16 22:17:18.123985 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:18.123950 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" event={"ID":"df5954e5-8adb-4e96-9b36-b19c37018858","Type":"ContainerStarted","Data":"ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2"} Jul 16 22:17:18.123985 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:18.123993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" event={"ID":"df5954e5-8adb-4e96-9b36-b19c37018858","Type":"ContainerStarted","Data":"2864a74388d9188a72ca45443eedc72e658f66286178eaa251f05aae1cf73b06"} Jul 16 22:17:18.143869 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:18.143808 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" podStartSLOduration=1.143785952 podStartE2EDuration="1.143785952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:18.143343981 +0000 UTC m=+2285.185568222" watchObservedRunningTime="2026-07-16 22:17:18.143785952 +0000 UTC m=+2285.186010191" Jul 16 22:17:27.906681 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:27.906631 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:27.907160 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:27.906694 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:27.919296 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:27.919265 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:28.187427 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:28.187332 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:17:50.117163 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:50.117121 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:50.217034 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:50.216967 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:57.917587 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:57.917548 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt"] Jul 16 22:17:57.936778 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:57.936701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:58.805955 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:58.805909 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5"] Jul 16 22:17:58.815768 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:17:58.815732 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:05.410091 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:05.409998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:12.939064 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:12.939013 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" podUID="48278cd8-746e-470d-a734-5fe3c0c40f0a" containerName="main" containerID="cri-o://340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba" gracePeriod=45 Jul 16 22:18:13.193988 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.193912 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:18:13.378552 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.378499 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-dshm\") pod \"48278cd8-746e-470d-a734-5fe3c0c40f0a\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " Jul 16 22:18:13.378757 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.378590 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-tmp-dir\") pod \"48278cd8-746e-470d-a734-5fe3c0c40f0a\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " Jul 16 22:18:13.378757 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.378641 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-home\") pod \"48278cd8-746e-470d-a734-5fe3c0c40f0a\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " Jul 16 22:18:13.378757 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.378666 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48278cd8-746e-470d-a734-5fe3c0c40f0a-tls-certs\") pod \"48278cd8-746e-470d-a734-5fe3c0c40f0a\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " Jul 16 22:18:13.378757 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.378698 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxbwx\" (UniqueName: \"kubernetes.io/projected/48278cd8-746e-470d-a734-5fe3c0c40f0a-kube-api-access-mxbwx\") pod \"48278cd8-746e-470d-a734-5fe3c0c40f0a\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " Jul 16 22:18:13.378964 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.378843 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-model-cache\") pod \"48278cd8-746e-470d-a734-5fe3c0c40f0a\" (UID: \"48278cd8-746e-470d-a734-5fe3c0c40f0a\") " Jul 16 22:18:13.378964 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.378902 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48278cd8-746e-470d-a734-5fe3c0c40f0a" (UID: "48278cd8-746e-470d-a734-5fe3c0c40f0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:18:13.378964 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.378930 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-home" (OuterVolumeSpecName: "home") pod "48278cd8-746e-470d-a734-5fe3c0c40f0a" (UID: "48278cd8-746e-470d-a734-5fe3c0c40f0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:18:13.379183 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.379161 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "48278cd8-746e-470d-a734-5fe3c0c40f0a" (UID: "48278cd8-746e-470d-a734-5fe3c0c40f0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:18:13.379246 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.379197 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:13.379246 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.379211 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:13.380842 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.380820 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48278cd8-746e-470d-a734-5fe3c0c40f0a-kube-api-access-mxbwx" (OuterVolumeSpecName: "kube-api-access-mxbwx") pod "48278cd8-746e-470d-a734-5fe3c0c40f0a" (UID: "48278cd8-746e-470d-a734-5fe3c0c40f0a"). InnerVolumeSpecName "kube-api-access-mxbwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:18:13.381193 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.381169 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-dshm" (OuterVolumeSpecName: "dshm") pod "48278cd8-746e-470d-a734-5fe3c0c40f0a" (UID: "48278cd8-746e-470d-a734-5fe3c0c40f0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:18:13.381193 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.381171 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48278cd8-746e-470d-a734-5fe3c0c40f0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48278cd8-746e-470d-a734-5fe3c0c40f0a" (UID: "48278cd8-746e-470d-a734-5fe3c0c40f0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:18:13.402929 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.402897 2573 generic.go:356] "Generic (PLEG): container finished" podID="48278cd8-746e-470d-a734-5fe3c0c40f0a" containerID="340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba" exitCode=0 Jul 16 22:18:13.403099 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.402936 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" Jul 16 22:18:13.403099 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.402962 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" event={"ID":"48278cd8-746e-470d-a734-5fe3c0c40f0a","Type":"ContainerDied","Data":"340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba"} Jul 16 22:18:13.403099 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.402993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt" event={"ID":"48278cd8-746e-470d-a734-5fe3c0c40f0a","Type":"ContainerDied","Data":"a3ff192e1cea4188e17cbffdbd3cf12fc380227a8152e0bce7ecca40bb47f8b9"} Jul 16 22:18:13.403099 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.403009 2573 scope.go:117] "RemoveContainer" containerID="340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba" Jul 16 22:18:13.412363 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.412337 2573 scope.go:117] "RemoveContainer" containerID="340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba" Jul 16 22:18:13.412681 ip-10-0-133-93 kubenswrapper[2573]: E0716 22:18:13.412663 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba\": container with ID starting with 340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba not found: ID does not exist" containerID="340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba" Jul 16 22:18:13.412759 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.412688 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba"} err="failed to get container status \"340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba\": rpc error: code = NotFound desc = could not find container \"340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba\": container with ID starting with 340299485c6241a7fe513eee5a44055e0049a3ba6f85bb0ed47131b049d377ba not found: ID does not exist" Jul 16 22:18:13.426627 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.426593 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt"] Jul 16 22:18:13.431161 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.431133 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5d61a3/sub-test-model-5d61a3-kserve-8df7588c9-nxkzt"] Jul 16 22:18:13.480003 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.479905 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:13.480003 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.479939 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48278cd8-746e-470d-a734-5fe3c0c40f0a-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:13.480003 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.479948 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxbwx\" (UniqueName: \"kubernetes.io/projected/48278cd8-746e-470d-a734-5fe3c0c40f0a-kube-api-access-mxbwx\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:13.480003 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.479957 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48278cd8-746e-470d-a734-5fe3c0c40f0a-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:13.541193 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.541157 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48278cd8-746e-470d-a734-5fe3c0c40f0a" path="/var/lib/kubelet/pods/48278cd8-746e-470d-a734-5fe3c0c40f0a/volumes" Jul 16 22:18:13.827443 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:13.827407 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" podUID="df5954e5-8adb-4e96-9b36-b19c37018858" containerName="main" containerID="cri-o://ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2" gracePeriod=45 Jul 16 22:18:14.092765 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.092689 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:18:14.187435 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187400 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-tmp-dir\") pod \"df5954e5-8adb-4e96-9b36-b19c37018858\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " Jul 16 22:18:14.187646 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187446 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-home\") pod \"df5954e5-8adb-4e96-9b36-b19c37018858\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " Jul 16 22:18:14.187646 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187492 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skwbh\" (UniqueName: \"kubernetes.io/projected/df5954e5-8adb-4e96-9b36-b19c37018858-kube-api-access-skwbh\") pod \"df5954e5-8adb-4e96-9b36-b19c37018858\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " Jul 16 22:18:14.187646 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187550 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-dshm\") pod \"df5954e5-8adb-4e96-9b36-b19c37018858\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " Jul 16 22:18:14.187646 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187591 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5954e5-8adb-4e96-9b36-b19c37018858-tls-certs\") pod \"df5954e5-8adb-4e96-9b36-b19c37018858\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " Jul 16 22:18:14.187646 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187610 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-model-cache\") pod \"df5954e5-8adb-4e96-9b36-b19c37018858\" (UID: \"df5954e5-8adb-4e96-9b36-b19c37018858\") " Jul 16 22:18:14.187945 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187781 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df5954e5-8adb-4e96-9b36-b19c37018858" (UID: "df5954e5-8adb-4e96-9b36-b19c37018858"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:18:14.187945 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187794 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-home" (OuterVolumeSpecName: "home") pod "df5954e5-8adb-4e96-9b36-b19c37018858" (UID: "df5954e5-8adb-4e96-9b36-b19c37018858"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:18:14.187945 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187908 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:14.187945 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187927 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:14.187945 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.187929 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-model-cache" (OuterVolumeSpecName: "model-cache") pod "df5954e5-8adb-4e96-9b36-b19c37018858" (UID: "df5954e5-8adb-4e96-9b36-b19c37018858"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:18:14.189887 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.189861 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5954e5-8adb-4e96-9b36-b19c37018858-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df5954e5-8adb-4e96-9b36-b19c37018858" (UID: "df5954e5-8adb-4e96-9b36-b19c37018858"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:18:14.190003 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.189918 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-dshm" (OuterVolumeSpecName: "dshm") pod "df5954e5-8adb-4e96-9b36-b19c37018858" (UID: "df5954e5-8adb-4e96-9b36-b19c37018858"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:18:14.190003 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.189929 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5954e5-8adb-4e96-9b36-b19c37018858-kube-api-access-skwbh" (OuterVolumeSpecName: "kube-api-access-skwbh") pod "df5954e5-8adb-4e96-9b36-b19c37018858" (UID: "df5954e5-8adb-4e96-9b36-b19c37018858"). InnerVolumeSpecName "kube-api-access-skwbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:18:14.288573 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.288513 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skwbh\" (UniqueName: \"kubernetes.io/projected/df5954e5-8adb-4e96-9b36-b19c37018858-kube-api-access-skwbh\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:14.288573 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.288564 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:14.288573 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.288576 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5954e5-8adb-4e96-9b36-b19c37018858-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:14.288573 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.288585 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5954e5-8adb-4e96-9b36-b19c37018858-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:18:14.409162 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.409066 2573 generic.go:356] "Generic (PLEG): container finished" podID="df5954e5-8adb-4e96-9b36-b19c37018858" containerID="ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2" exitCode=0 Jul 16 22:18:14.409162 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.409112 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" Jul 16 22:18:14.409162 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.409148 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" event={"ID":"df5954e5-8adb-4e96-9b36-b19c37018858","Type":"ContainerDied","Data":"ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2"} Jul 16 22:18:14.409441 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.409191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5" event={"ID":"df5954e5-8adb-4e96-9b36-b19c37018858","Type":"ContainerDied","Data":"2864a74388d9188a72ca45443eedc72e658f66286178eaa251f05aae1cf73b06"} Jul 16 22:18:14.409441 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.409211 2573 scope.go:117] "RemoveContainer" containerID="ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2" Jul 16 22:18:14.419624 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.419597 2573 scope.go:117] "RemoveContainer" containerID="ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2" Jul 16 22:18:14.419917 ip-10-0-133-93 kubenswrapper[2573]: E0716 22:18:14.419896 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2\": container with ID starting with ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2 not found: ID does not exist" containerID="ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2" Jul 16 22:18:14.419976 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.419926 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2"} err="failed to get container status \"ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2\": rpc error: code = NotFound desc = could not find container \"ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2\": container with ID starting with ae24779e832d23aa1c3ac1ecdde7ba7bd47c52849a1fe413d7d9552d286497d2 not found: ID does not exist" Jul 16 22:18:14.431615 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.431587 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5"] Jul 16 22:18:14.435721 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:14.435694 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/sub-test-model-ff5f80-kserve-8467d7f997-bc5v5"] Jul 16 22:18:15.542179 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:15.542143 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5954e5-8adb-4e96-9b36-b19c37018858" path="/var/lib/kubelet/pods/df5954e5-8adb-4e96-9b36-b19c37018858/volumes" Jul 16 22:18:38.790934 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.790895 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2"] Jul 16 22:18:38.791403 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.791319 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5954e5-8adb-4e96-9b36-b19c37018858" containerName="main" Jul 16 22:18:38.791403 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.791336 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5954e5-8adb-4e96-9b36-b19c37018858" containerName="main" Jul 16 22:18:38.791403 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.791368 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48278cd8-746e-470d-a734-5fe3c0c40f0a" containerName="main" Jul 16 22:18:38.791403 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.791374 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="48278cd8-746e-470d-a734-5fe3c0c40f0a" containerName="main" Jul 16 22:18:38.791559 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.791436 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5954e5-8adb-4e96-9b36-b19c37018858" containerName="main" Jul 16 22:18:38.791559 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.791446 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="48278cd8-746e-470d-a734-5fe3c0c40f0a" containerName="main" Jul 16 22:18:38.794720 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.794695 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:38.798369 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.798348 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff5f80\"/\"kube-root-ca.crt\"" Jul 16 22:18:38.798512 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.798351 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff5f80\"/\"rate-test-model-ff5f80-kserve-self-signed-certs\"" Jul 16 22:18:38.798512 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.798428 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff5f80\"/\"default-dockercfg-nk4tz\"" Jul 16 22:18:38.799924 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.799911 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff5f80\"/\"openshift-service-ca.crt\"" Jul 16 22:18:38.806089 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.806065 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2"] Jul 16 22:18:38.817602 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.817569 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:38.939807 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.939752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-dshm\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:38.939807 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.939808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84461e8-d240-43bf-8496-8038daee7d5e-tls-certs\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:38.940092 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.939915 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-home\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:38.940092 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.939950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-model-cache\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:38.940092 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.939987 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5d5n\" (UniqueName: \"kubernetes.io/projected/c84461e8-d240-43bf-8496-8038daee7d5e-kube-api-access-j5d5n\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:38.940092 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:38.940032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-tmp-dir\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.040675 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.040634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-dshm\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.040675 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.040675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84461e8-d240-43bf-8496-8038daee7d5e-tls-certs\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.040997 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.040823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-home\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.040997 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.040854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-model-cache\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.040997 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.040882 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5d5n\" (UniqueName: \"kubernetes.io/projected/c84461e8-d240-43bf-8496-8038daee7d5e-kube-api-access-j5d5n\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.040997 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.040926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-tmp-dir\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.041303 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.041248 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-model-cache\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.041303 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.041278 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-home\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.041408 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.041310 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-tmp-dir\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.043187 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.043153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-dshm\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.043339 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.043321 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84461e8-d240-43bf-8496-8038daee7d5e-tls-certs\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.049706 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.049683 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5d5n\" (UniqueName: \"kubernetes.io/projected/c84461e8-d240-43bf-8496-8038daee7d5e-kube-api-access-j5d5n\") pod \"rate-test-model-ff5f80-kserve-95dff787f-fddc2\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.106606 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.106561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:39.247195 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.247165 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2"] Jul 16 22:18:39.551747 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.551643 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" event={"ID":"c84461e8-d240-43bf-8496-8038daee7d5e","Type":"ContainerStarted","Data":"a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660"} Jul 16 22:18:39.551747 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.551689 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" event={"ID":"c84461e8-d240-43bf-8496-8038daee7d5e","Type":"ContainerStarted","Data":"6927e8b1fe194c9bc5b07a416d0faa2a60c2ce70afec75955769ea1a17d6614e"} Jul 16 22:18:39.570290 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:39.570232 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" podStartSLOduration=1.570212785 podStartE2EDuration="1.570212785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:39.568636238 +0000 UTC m=+2366.610860477" watchObservedRunningTime="2026-07-16 22:18:39.570212785 +0000 UTC m=+2366.612437022" Jul 16 22:18:40.075424 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:40.075389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:40.179926 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:40.179881 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:49.106736 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:49.106689 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:49.107119 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:49.106760 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:49.119727 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:49.119696 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:49.280312 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:49.280271 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:49.381888 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:49.381802 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:49.613642 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:49.613603 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:18:51.378939 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:18:51.378899 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:13.498415 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:13.498378 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:25.248555 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:25.248493 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:26.282000 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:26.281964 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2"] Jul 16 22:19:26.304029 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:26.304002 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:27.211570 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.211529 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7573-140428995260080"] Jul 16 22:19:27.215079 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.215062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428995260080" Jul 16 22:19:27.217482 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.217458 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-szfpx\"" Jul 16 22:19:27.220532 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.220497 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428995260080"] Jul 16 22:19:27.314450 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.314402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhsn\" (UniqueName: \"kubernetes.io/projected/2af0499f-a007-482d-b5ff-5c430c100a84-kube-api-access-srhsn\") pod \"test-curl-7573-140428995260080\" (UID: \"2af0499f-a007-482d-b5ff-5c430c100a84\") " pod="openshift-ingress/test-curl-7573-140428995260080" Jul 16 22:19:27.415023 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.414985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhsn\" (UniqueName: \"kubernetes.io/projected/2af0499f-a007-482d-b5ff-5c430c100a84-kube-api-access-srhsn\") pod \"test-curl-7573-140428995260080\" (UID: \"2af0499f-a007-482d-b5ff-5c430c100a84\") " pod="openshift-ingress/test-curl-7573-140428995260080" Jul 16 22:19:27.423757 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.423717 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhsn\" (UniqueName: \"kubernetes.io/projected/2af0499f-a007-482d-b5ff-5c430c100a84-kube-api-access-srhsn\") pod \"test-curl-7573-140428995260080\" (UID: \"2af0499f-a007-482d-b5ff-5c430c100a84\") " pod="openshift-ingress/test-curl-7573-140428995260080" Jul 16 22:19:27.525283 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.525189 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428995260080" Jul 16 22:19:27.665796 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.665733 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428995260080"] Jul 16 22:19:27.668725 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:19:27.668696 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2af0499f_a007_482d_b5ff_5c430c100a84.slice/crio-262a8f208f41aea99aec9bed26de25f221519c30cfb6f7af646f093cc443c284 WatchSource:0}: Error finding container 262a8f208f41aea99aec9bed26de25f221519c30cfb6f7af646f093cc443c284: Status 404 returned error can't find the container with id 262a8f208f41aea99aec9bed26de25f221519c30cfb6f7af646f093cc443c284 Jul 16 22:19:27.670467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.670453 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 22:19:27.802412 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:27.802378 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428995260080" event={"ID":"2af0499f-a007-482d-b5ff-5c430c100a84","Type":"ContainerStarted","Data":"262a8f208f41aea99aec9bed26de25f221519c30cfb6f7af646f093cc443c284"} Jul 16 22:19:28.808710 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:28.808674 2573 generic.go:356] "Generic (PLEG): container finished" podID="2af0499f-a007-482d-b5ff-5c430c100a84" containerID="925ab1285e56eec9b14daeb1fb7e31552a5d4eff7f811c2b31dbf618d7bae6ab" exitCode=0 Jul 16 22:19:28.809101 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:28.808760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428995260080" event={"ID":"2af0499f-a007-482d-b5ff-5c430c100a84","Type":"ContainerDied","Data":"925ab1285e56eec9b14daeb1fb7e31552a5d4eff7f811c2b31dbf618d7bae6ab"} Jul 16 22:19:29.956076 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:29.956050 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428995260080" Jul 16 22:19:29.978890 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:29.978865 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7573-140428995260080_2af0499f-a007-482d-b5ff-5c430c100a84/test-curl-7573-140428995260080/0.log" Jul 16 22:19:30.000767 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.000735 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7573-140428995260080"] Jul 16 22:19:30.005098 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.005074 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7573-140428995260080"] Jul 16 22:19:30.040829 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.040794 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srhsn\" (UniqueName: \"kubernetes.io/projected/2af0499f-a007-482d-b5ff-5c430c100a84-kube-api-access-srhsn\") pod \"2af0499f-a007-482d-b5ff-5c430c100a84\" (UID: \"2af0499f-a007-482d-b5ff-5c430c100a84\") " Jul 16 22:19:30.043049 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.043017 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af0499f-a007-482d-b5ff-5c430c100a84-kube-api-access-srhsn" (OuterVolumeSpecName: "kube-api-access-srhsn") pod "2af0499f-a007-482d-b5ff-5c430c100a84" (UID: "2af0499f-a007-482d-b5ff-5c430c100a84"). InnerVolumeSpecName "kube-api-access-srhsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:19:30.130900 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.130806 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7573-140428995258672"] Jul 16 22:19:30.131320 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.131300 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af0499f-a007-482d-b5ff-5c430c100a84" containerName="test-curl-7573-140428995260080" Jul 16 22:19:30.131413 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.131322 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af0499f-a007-482d-b5ff-5c430c100a84" containerName="test-curl-7573-140428995260080" Jul 16 22:19:30.131467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.131430 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af0499f-a007-482d-b5ff-5c430c100a84" containerName="test-curl-7573-140428995260080" Jul 16 22:19:30.134550 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.134529 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428995258672" Jul 16 22:19:30.141713 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.141681 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7d9b\" (UniqueName: \"kubernetes.io/projected/ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d-kube-api-access-q7d9b\") pod \"test-curl-7573-140428995258672\" (UID: \"ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d\") " pod="openshift-ingress/test-curl-7573-140428995258672" Jul 16 22:19:30.141898 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.141882 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhsn\" (UniqueName: \"kubernetes.io/projected/2af0499f-a007-482d-b5ff-5c430c100a84-kube-api-access-srhsn\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:30.142012 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.141983 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428995258672"] Jul 16 22:19:30.242605 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.242567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7d9b\" (UniqueName: \"kubernetes.io/projected/ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d-kube-api-access-q7d9b\") pod \"test-curl-7573-140428995258672\" (UID: \"ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d\") " pod="openshift-ingress/test-curl-7573-140428995258672" Jul 16 22:19:30.250887 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.250857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7d9b\" (UniqueName: \"kubernetes.io/projected/ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d-kube-api-access-q7d9b\") pod \"test-curl-7573-140428995258672\" (UID: \"ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d\") " pod="openshift-ingress/test-curl-7573-140428995258672" Jul 16 22:19:30.445577 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.445464 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428995258672" Jul 16 22:19:30.586312 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.586276 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428995258672"] Jul 16 22:19:30.589511 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:19:30.589474 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded37dd5b_9bd1_43cb_873d_1d9bde5a1d9d.slice/crio-b42734db0c068405b51dbdcb831de47f2f5105ae215875daf7b7d4deb289f7ff WatchSource:0}: Error finding container b42734db0c068405b51dbdcb831de47f2f5105ae215875daf7b7d4deb289f7ff: Status 404 returned error can't find the container with id b42734db0c068405b51dbdcb831de47f2f5105ae215875daf7b7d4deb289f7ff Jul 16 22:19:30.819924 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.819894 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428995260080" Jul 16 22:19:30.820123 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.819905 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262a8f208f41aea99aec9bed26de25f221519c30cfb6f7af646f093cc443c284" Jul 16 22:19:30.828311 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:30.828277 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428995258672" event={"ID":"ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d","Type":"ContainerStarted","Data":"b42734db0c068405b51dbdcb831de47f2f5105ae215875daf7b7d4deb289f7ff"} Jul 16 22:19:30.834437 ip-10-0-133-93 kubenswrapper[2573]: E0716 22:19:30.834402 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7573-140428995260080\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="2af0499f-a007-482d-b5ff-5c430c100a84" pod="openshift-ingress/test-curl-7573-140428995260080" Jul 16 22:19:31.541801 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:31.541759 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af0499f-a007-482d-b5ff-5c430c100a84" path="/var/lib/kubelet/pods/2af0499f-a007-482d-b5ff-5c430c100a84/volumes" Jul 16 22:19:31.835208 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:31.835176 2573 generic.go:356] "Generic (PLEG): container finished" podID="ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d" containerID="802790dc869e0be9863def3e254fe42f4ddda7c777208985a8def58489e03f95" exitCode=0 Jul 16 22:19:31.835403 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:31.835237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428995258672" event={"ID":"ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d","Type":"ContainerDied","Data":"802790dc869e0be9863def3e254fe42f4ddda7c777208985a8def58489e03f95"} Jul 16 22:19:32.983259 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:32.983228 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428995258672" Jul 16 22:19:33.004186 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.004157 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7573-140428995258672_ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d/test-curl-7573-140428995258672/0.log" Jul 16 22:19:33.030646 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.030605 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7573-140428995258672"] Jul 16 22:19:33.035365 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.035321 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7573-140428995258672"] Jul 16 22:19:33.069638 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.069603 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7d9b\" (UniqueName: \"kubernetes.io/projected/ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d-kube-api-access-q7d9b\") pod \"ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d\" (UID: \"ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d\") " Jul 16 22:19:33.071775 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.071742 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d-kube-api-access-q7d9b" (OuterVolumeSpecName: "kube-api-access-q7d9b") pod "ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d" (UID: "ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d"). InnerVolumeSpecName "kube-api-access-q7d9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:19:33.171132 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.171036 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7d9b\" (UniqueName: \"kubernetes.io/projected/ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d-kube-api-access-q7d9b\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:33.541778 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.541684 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d" path="/var/lib/kubelet/pods/ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d/volumes" Jul 16 22:19:33.845701 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.845672 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428995258672" Jul 16 22:19:33.845893 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:33.845672 2573 scope.go:117] "RemoveContainer" containerID="802790dc869e0be9863def3e254fe42f4ddda7c777208985a8def58489e03f95" Jul 16 22:19:36.395429 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.395349 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7573-140428994111360"] Jul 16 22:19:36.395830 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.395817 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d" containerName="test-curl-7573-140428995258672" Jul 16 22:19:36.395901 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.395833 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d" containerName="test-curl-7573-140428995258672" Jul 16 22:19:36.395901 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.395896 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed37dd5b-9bd1-43cb-873d-1d9bde5a1d9d" containerName="test-curl-7573-140428995258672" Jul 16 22:19:36.400242 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.400225 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994111360" Jul 16 22:19:36.403173 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.403151 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-szfpx\"" Jul 16 22:19:36.405090 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.405070 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428994111360"] Jul 16 22:19:36.510581 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.507861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9vt\" (UniqueName: \"kubernetes.io/projected/b52c8f06-3b04-427a-98e2-2eef1219477b-kube-api-access-qq9vt\") pod \"test-curl-7573-140428994111360\" (UID: \"b52c8f06-3b04-427a-98e2-2eef1219477b\") " pod="openshift-ingress/test-curl-7573-140428994111360" Jul 16 22:19:36.608483 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.608446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9vt\" (UniqueName: \"kubernetes.io/projected/b52c8f06-3b04-427a-98e2-2eef1219477b-kube-api-access-qq9vt\") pod \"test-curl-7573-140428994111360\" (UID: \"b52c8f06-3b04-427a-98e2-2eef1219477b\") " pod="openshift-ingress/test-curl-7573-140428994111360" Jul 16 22:19:36.617037 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.617004 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9vt\" (UniqueName: \"kubernetes.io/projected/b52c8f06-3b04-427a-98e2-2eef1219477b-kube-api-access-qq9vt\") pod \"test-curl-7573-140428994111360\" (UID: \"b52c8f06-3b04-427a-98e2-2eef1219477b\") " pod="openshift-ingress/test-curl-7573-140428994111360" Jul 16 22:19:36.709875 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.709781 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994111360" Jul 16 22:19:36.842702 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.842657 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428994111360"] Jul 16 22:19:36.845004 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:19:36.844971 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb52c8f06_3b04_427a_98e2_2eef1219477b.slice/crio-21b23375f999ddcef347ef4ed857ace9ecb942adc2795239570fae33ec66fc5d WatchSource:0}: Error finding container 21b23375f999ddcef347ef4ed857ace9ecb942adc2795239570fae33ec66fc5d: Status 404 returned error can't find the container with id 21b23375f999ddcef347ef4ed857ace9ecb942adc2795239570fae33ec66fc5d Jul 16 22:19:36.863063 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:36.863028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428994111360" event={"ID":"b52c8f06-3b04-427a-98e2-2eef1219477b","Type":"ContainerStarted","Data":"21b23375f999ddcef347ef4ed857ace9ecb942adc2795239570fae33ec66fc5d"} Jul 16 22:19:37.869545 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:37.869486 2573 generic.go:356] "Generic (PLEG): container finished" podID="b52c8f06-3b04-427a-98e2-2eef1219477b" containerID="ef0559073fc57fadac6a5ac1bfbb16c2eacdd79d0cf2596a9a2871f99d6f00bc" exitCode=0 Jul 16 22:19:37.869943 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:37.869576 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428994111360" event={"ID":"b52c8f06-3b04-427a-98e2-2eef1219477b","Type":"ContainerDied","Data":"ef0559073fc57fadac6a5ac1bfbb16c2eacdd79d0cf2596a9a2871f99d6f00bc"} Jul 16 22:19:39.017594 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.017567 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994111360" Jul 16 22:19:39.039335 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.039288 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7573-140428994111360_b52c8f06-3b04-427a-98e2-2eef1219477b/test-curl-7573-140428994111360/0.log" Jul 16 22:19:39.063583 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.063549 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7573-140428994111360"] Jul 16 22:19:39.069484 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.069459 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7573-140428994111360"] Jul 16 22:19:39.134251 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.134142 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq9vt\" (UniqueName: \"kubernetes.io/projected/b52c8f06-3b04-427a-98e2-2eef1219477b-kube-api-access-qq9vt\") pod \"b52c8f06-3b04-427a-98e2-2eef1219477b\" (UID: \"b52c8f06-3b04-427a-98e2-2eef1219477b\") " Jul 16 22:19:39.136683 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.136646 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52c8f06-3b04-427a-98e2-2eef1219477b-kube-api-access-qq9vt" (OuterVolumeSpecName: "kube-api-access-qq9vt") pod "b52c8f06-3b04-427a-98e2-2eef1219477b" (UID: "b52c8f06-3b04-427a-98e2-2eef1219477b"). InnerVolumeSpecName "kube-api-access-qq9vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:19:39.190045 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.190010 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:39.190434 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.190421 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52c8f06-3b04-427a-98e2-2eef1219477b" containerName="test-curl-7573-140428994111360" Jul 16 22:19:39.190478 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.190436 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52c8f06-3b04-427a-98e2-2eef1219477b" containerName="test-curl-7573-140428994111360" Jul 16 22:19:39.190541 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.190514 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52c8f06-3b04-427a-98e2-2eef1219477b" containerName="test-curl-7573-140428994111360" Jul 16 22:19:39.193689 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.193667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:39.200292 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.200266 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:39.235568 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.235537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6d77\" (UniqueName: \"kubernetes.io/projected/6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19-kube-api-access-q6d77\") pod \"test-curl-7573-140428994112368\" (UID: \"6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19\") " pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:39.235724 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.235649 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq9vt\" (UniqueName: \"kubernetes.io/projected/b52c8f06-3b04-427a-98e2-2eef1219477b-kube-api-access-qq9vt\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:39.337163 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.337123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6d77\" (UniqueName: \"kubernetes.io/projected/6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19-kube-api-access-q6d77\") pod \"test-curl-7573-140428994112368\" (UID: \"6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19\") " pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:39.346726 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.346692 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6d77\" (UniqueName: \"kubernetes.io/projected/6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19-kube-api-access-q6d77\") pod \"test-curl-7573-140428994112368\" (UID: \"6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19\") " pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:39.504310 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.504211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:39.542315 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.542276 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b52c8f06-3b04-427a-98e2-2eef1219477b" path="/var/lib/kubelet/pods/b52c8f06-3b04-427a-98e2-2eef1219477b/volumes" Jul 16 22:19:39.637922 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.637891 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:39.638794 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:19:39.638769 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e9cf1a1_cf53_4260_ab2c_ebaa2a5e5b19.slice/crio-3bb98ae95cb0a4a498ba29d310cffafa7b63d0ecbfc7c3b2e16926dce47353f5 WatchSource:0}: Error finding container 3bb98ae95cb0a4a498ba29d310cffafa7b63d0ecbfc7c3b2e16926dce47353f5: Status 404 returned error can't find the container with id 3bb98ae95cb0a4a498ba29d310cffafa7b63d0ecbfc7c3b2e16926dce47353f5 Jul 16 22:19:39.880533 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.880484 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994111360" Jul 16 22:19:39.880533 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.880496 2573 scope.go:117] "RemoveContainer" containerID="ef0559073fc57fadac6a5ac1bfbb16c2eacdd79d0cf2596a9a2871f99d6f00bc" Jul 16 22:19:39.881828 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:39.881715 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428994112368" event={"ID":"6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19","Type":"ContainerStarted","Data":"3bb98ae95cb0a4a498ba29d310cffafa7b63d0ecbfc7c3b2e16926dce47353f5"} Jul 16 22:19:40.888315 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:40.888278 2573 generic.go:356] "Generic (PLEG): container finished" podID="6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19" containerID="3c53398d9255d6ad701af2363f44162fbee7aef52ecc171c9a260d6ca1e51708" exitCode=0 Jul 16 22:19:40.888729 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:40.888362 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428994112368" event={"ID":"6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19","Type":"ContainerDied","Data":"3c53398d9255d6ad701af2363f44162fbee7aef52ecc171c9a260d6ca1e51708"} Jul 16 22:19:41.303330 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.303267 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" podUID="c84461e8-d240-43bf-8496-8038daee7d5e" containerName="main" containerID="cri-o://a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660" gracePeriod=45 Jul 16 22:19:41.588103 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.588079 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:19:41.656943 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.656909 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-tmp-dir\") pod \"c84461e8-d240-43bf-8496-8038daee7d5e\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " Jul 16 22:19:41.657128 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.656965 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84461e8-d240-43bf-8496-8038daee7d5e-tls-certs\") pod \"c84461e8-d240-43bf-8496-8038daee7d5e\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " Jul 16 22:19:41.657128 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657011 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5d5n\" (UniqueName: \"kubernetes.io/projected/c84461e8-d240-43bf-8496-8038daee7d5e-kube-api-access-j5d5n\") pod \"c84461e8-d240-43bf-8496-8038daee7d5e\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " Jul 16 22:19:41.657128 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657051 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-home\") pod \"c84461e8-d240-43bf-8496-8038daee7d5e\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " Jul 16 22:19:41.657128 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657080 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-model-cache\") pod \"c84461e8-d240-43bf-8496-8038daee7d5e\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " Jul 16 22:19:41.657128 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657109 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-dshm\") pod \"c84461e8-d240-43bf-8496-8038daee7d5e\" (UID: \"c84461e8-d240-43bf-8496-8038daee7d5e\") " Jul 16 22:19:41.657445 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657231 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c84461e8-d240-43bf-8496-8038daee7d5e" (UID: "c84461e8-d240-43bf-8496-8038daee7d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:19:41.657445 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657356 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-home" (OuterVolumeSpecName: "home") pod "c84461e8-d240-43bf-8496-8038daee7d5e" (UID: "c84461e8-d240-43bf-8496-8038daee7d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:19:41.657445 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657395 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c84461e8-d240-43bf-8496-8038daee7d5e" (UID: "c84461e8-d240-43bf-8496-8038daee7d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:19:41.657598 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657582 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:41.657642 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657600 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:41.657642 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.657610 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:41.659203 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.659178 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-dshm" (OuterVolumeSpecName: "dshm") pod "c84461e8-d240-43bf-8496-8038daee7d5e" (UID: "c84461e8-d240-43bf-8496-8038daee7d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:19:41.659693 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.659671 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84461e8-d240-43bf-8496-8038daee7d5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c84461e8-d240-43bf-8496-8038daee7d5e" (UID: "c84461e8-d240-43bf-8496-8038daee7d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:19:41.659765 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.659677 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84461e8-d240-43bf-8496-8038daee7d5e-kube-api-access-j5d5n" (OuterVolumeSpecName: "kube-api-access-j5d5n") pod "c84461e8-d240-43bf-8496-8038daee7d5e" (UID: "c84461e8-d240-43bf-8496-8038daee7d5e"). InnerVolumeSpecName "kube-api-access-j5d5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:19:41.758412 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.758373 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84461e8-d240-43bf-8496-8038daee7d5e-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:41.758412 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.758411 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5d5n\" (UniqueName: \"kubernetes.io/projected/c84461e8-d240-43bf-8496-8038daee7d5e-kube-api-access-j5d5n\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:41.758412 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.758425 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84461e8-d240-43bf-8496-8038daee7d5e-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:41.895748 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.895659 2573 generic.go:356] "Generic (PLEG): container finished" podID="c84461e8-d240-43bf-8496-8038daee7d5e" containerID="a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660" exitCode=0 Jul 16 22:19:41.895748 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.895702 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" Jul 16 22:19:41.896250 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.895744 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" event={"ID":"c84461e8-d240-43bf-8496-8038daee7d5e","Type":"ContainerDied","Data":"a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660"} Jul 16 22:19:41.896250 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.895796 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2" event={"ID":"c84461e8-d240-43bf-8496-8038daee7d5e","Type":"ContainerDied","Data":"6927e8b1fe194c9bc5b07a416d0faa2a60c2ce70afec75955769ea1a17d6614e"} Jul 16 22:19:41.896250 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.895819 2573 scope.go:117] "RemoveContainer" containerID="a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660" Jul 16 22:19:41.905099 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.905077 2573 scope.go:117] "RemoveContainer" containerID="a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660" Jul 16 22:19:41.905359 ip-10-0-133-93 kubenswrapper[2573]: E0716 22:19:41.905337 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660\": container with ID starting with a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660 not found: ID does not exist" containerID="a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660" Jul 16 22:19:41.905428 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.905372 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660"} err="failed to get container status \"a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660\": rpc error: code = NotFound desc = could not find container \"a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660\": container with ID starting with a22159725bb37e3e7598b82463a0f75883b333e34fced90fc128a1b486330660 not found: ID does not exist" Jul 16 22:19:41.918547 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.918499 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2"] Jul 16 22:19:41.922010 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:41.921979 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ff5f80/rate-test-model-ff5f80-kserve-95dff787f-fddc2"] Jul 16 22:19:42.036264 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.036239 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:42.057387 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.057328 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7573-140428994112368_6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19/test-curl-7573-140428994112368/0.log" Jul 16 22:19:42.080499 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.080467 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:42.085044 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.085015 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:42.162727 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.162640 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6d77\" (UniqueName: \"kubernetes.io/projected/6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19-kube-api-access-q6d77\") pod \"6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19\" (UID: \"6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19\") " Jul 16 22:19:42.164700 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.164673 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19-kube-api-access-q6d77" (OuterVolumeSpecName: "kube-api-access-q6d77") pod "6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19" (UID: "6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19"). InnerVolumeSpecName "kube-api-access-q6d77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:19:42.223492 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.223456 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:42.223871 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.223858 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19" containerName="test-curl-7573-140428994112368" Jul 16 22:19:42.223915 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.223873 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19" containerName="test-curl-7573-140428994112368" Jul 16 22:19:42.223915 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.223886 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84461e8-d240-43bf-8496-8038daee7d5e" containerName="main" Jul 16 22:19:42.223915 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.223892 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84461e8-d240-43bf-8496-8038daee7d5e" containerName="main" Jul 16 22:19:42.224013 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.223950 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19" containerName="test-curl-7573-140428994112368" Jul 16 22:19:42.224013 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.223959 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84461e8-d240-43bf-8496-8038daee7d5e" containerName="main" Jul 16 22:19:42.226957 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.226941 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:42.246801 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.246779 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:42.263389 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.263355 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6d77\" (UniqueName: \"kubernetes.io/projected/6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19-kube-api-access-q6d77\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:42.364846 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.364808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnbc\" (UniqueName: \"kubernetes.io/projected/be9745b6-5195-4fdb-8408-9a490d28515c-kube-api-access-rjnbc\") pod \"test-curl-7573-140428994112368\" (UID: \"be9745b6-5195-4fdb-8408-9a490d28515c\") " pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:42.466175 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.466078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnbc\" (UniqueName: \"kubernetes.io/projected/be9745b6-5195-4fdb-8408-9a490d28515c-kube-api-access-rjnbc\") pod \"test-curl-7573-140428994112368\" (UID: \"be9745b6-5195-4fdb-8408-9a490d28515c\") " pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:42.474113 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.474086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnbc\" (UniqueName: \"kubernetes.io/projected/be9745b6-5195-4fdb-8408-9a490d28515c-kube-api-access-rjnbc\") pod \"test-curl-7573-140428994112368\" (UID: \"be9745b6-5195-4fdb-8408-9a490d28515c\") " pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:42.535494 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.535448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:42.670072 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.670039 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:42.673849 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:19:42.673818 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe9745b6_5195_4fdb_8408_9a490d28515c.slice/crio-539fb82caae7aa71180b1d40959742d205f0c03f01805a4e956b6b48c1228801 WatchSource:0}: Error finding container 539fb82caae7aa71180b1d40959742d205f0c03f01805a4e956b6b48c1228801: Status 404 returned error can't find the container with id 539fb82caae7aa71180b1d40959742d205f0c03f01805a4e956b6b48c1228801 Jul 16 22:19:42.902139 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.902111 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:42.902139 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.902129 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bb98ae95cb0a4a498ba29d310cffafa7b63d0ecbfc7c3b2e16926dce47353f5" Jul 16 22:19:42.904138 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.904115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428994112368" event={"ID":"be9745b6-5195-4fdb-8408-9a490d28515c","Type":"ContainerStarted","Data":"539fb82caae7aa71180b1d40959742d205f0c03f01805a4e956b6b48c1228801"} Jul 16 22:19:42.914479 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:42.914449 2573 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7573-140428994112368" oldPodUID="6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19" podUID="be9745b6-5195-4fdb-8408-9a490d28515c" Jul 16 22:19:43.546416 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:43.546380 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19" path="/var/lib/kubelet/pods/6e9cf1a1-cf53-4260-ab2c-ebaa2a5e5b19/volumes" Jul 16 22:19:43.546908 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:43.546890 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c84461e8-d240-43bf-8496-8038daee7d5e" path="/var/lib/kubelet/pods/c84461e8-d240-43bf-8496-8038daee7d5e/volumes" Jul 16 22:19:43.910376 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:43.910288 2573 generic.go:356] "Generic (PLEG): container finished" podID="be9745b6-5195-4fdb-8408-9a490d28515c" containerID="dfb0fa362b51f36177d6b394a8af4627c5c71d13c15f71cc0163f81aea590712" exitCode=0 Jul 16 22:19:43.910376 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:43.910359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7573-140428994112368" event={"ID":"be9745b6-5195-4fdb-8408-9a490d28515c","Type":"ContainerDied","Data":"dfb0fa362b51f36177d6b394a8af4627c5c71d13c15f71cc0163f81aea590712"} Jul 16 22:19:45.058221 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.058196 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:45.079906 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.079875 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7573-140428994112368_be9745b6-5195-4fdb-8408-9a490d28515c/test-curl-7573-140428994112368/0.log" Jul 16 22:19:45.105880 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.105852 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:45.109810 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.109784 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7573-140428994112368"] Jul 16 22:19:45.192458 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.192363 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjnbc\" (UniqueName: \"kubernetes.io/projected/be9745b6-5195-4fdb-8408-9a490d28515c-kube-api-access-rjnbc\") pod \"be9745b6-5195-4fdb-8408-9a490d28515c\" (UID: \"be9745b6-5195-4fdb-8408-9a490d28515c\") " Jul 16 22:19:45.194445 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.194418 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9745b6-5195-4fdb-8408-9a490d28515c-kube-api-access-rjnbc" (OuterVolumeSpecName: "kube-api-access-rjnbc") pod "be9745b6-5195-4fdb-8408-9a490d28515c" (UID: "be9745b6-5195-4fdb-8408-9a490d28515c"). InnerVolumeSpecName "kube-api-access-rjnbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:19:45.293449 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.293408 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjnbc\" (UniqueName: \"kubernetes.io/projected/be9745b6-5195-4fdb-8408-9a490d28515c-kube-api-access-rjnbc\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\"" Jul 16 22:19:45.441349 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.441310 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:45.541548 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.541444 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be9745b6-5195-4fdb-8408-9a490d28515c" path="/var/lib/kubelet/pods/be9745b6-5195-4fdb-8408-9a490d28515c/volumes" Jul 16 22:19:45.813303 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.813237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:45.914114 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.914073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:45.921863 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.921840 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7573-140428994112368" Jul 16 22:19:45.922040 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:45.921881 2573 scope.go:117] "RemoveContainer" containerID="dfb0fa362b51f36177d6b394a8af4627c5c71d13c15f71cc0163f81aea590712" Jul 16 22:19:46.023461 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:46.023424 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:46.216512 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:46.216428 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:51.828069 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:51.828029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:51.925291 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:51.925242 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:52.094162 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:52.094051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:52.305066 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:52.305033 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:52.540937 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:52.540846 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:52.884723 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:52.884693 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:53.027757 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:53.027721 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:53.136823 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:53.136739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:53.302217 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:53.302182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:58.812134 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:58.812099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:58.912487 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:58.912451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:59.090847 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:59.090760 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:59.300112 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:19:59.300050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:20:04.981278 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:04.981248 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-db5748bcb-6vwzm_67a5213d-5613-4946-a352-8ae8f2443346/copy-manifests/0.log" Jul 16 22:20:04.988948 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:04.988922 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-db5748bcb-6vwzm_67a5213d-5613-4946-a352-8ae8f2443346/manager/0.log" Jul 16 22:20:07.558489 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:07.558458 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-mzrbb_68af045e-6189-4916-97f2-9959ad046053/manager/0.log" Jul 16 22:20:08.446870 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:08.446834 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b5c5c797-ztsd5_681b65a0-b05d-43c0-8ebf-8e1483b2bd26/kube-auth-proxy/0.log" Jul 16 22:20:08.809697 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:08.809664 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vmmsf_a5e5398f-c17b-405e-8855-67a8fdbb929c/payload-processing/0.log" Jul 16 22:20:09.486444 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:09.486415 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wj4s8_23352deb-b450-4b5d-82de-3f5c1e4de8ee/main/0.log" Jul 16 22:20:09.710428 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:09.710397 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-x4lsv_6b13a4c5-ceaf-4561-a854-f10942a0369f/main/0.log" Jul 16 22:20:09.819721 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:09.819691 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-cnc65_31c41c1f-05a3-40d6-ba9c-e1f077604386/main/0.log" Jul 16 22:20:13.798813 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:13.798765 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:20:14.875295 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:14.875257 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" limit=1024 Jul 16 22:20:16.803720 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:16.803687 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-db75d_38526ca8-ee16-4dfe-84d6-965b4546ab8f/global-pull-secret-syncer/0.log" Jul 16 22:20:16.944434 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:16.944389 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7ffsb_b07dc994-256a-44d4-8405-715a10a1feb6/konnectivity-agent/0.log" Jul 16 22:20:17.046137 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:17.046103 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-93.ec2.internal_ad611d48b80ded02db93237655788291/haproxy/0.log" Jul 16 22:20:23.013789 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:23.013747 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-mzrbb_68af045e-6189-4916-97f2-9959ad046053/manager/0.log" Jul 16 22:20:24.555332 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.555299 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_042787c0-eece-41ee-b1ad-12bcd75922f4/alertmanager/0.log" Jul 16 22:20:24.581036 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.581006 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_042787c0-eece-41ee-b1ad-12bcd75922f4/config-reloader/0.log" Jul 16 22:20:24.604610 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.604490 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_042787c0-eece-41ee-b1ad-12bcd75922f4/kube-rbac-proxy-web/0.log" Jul 16 22:20:24.627808 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.627783 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_042787c0-eece-41ee-b1ad-12bcd75922f4/kube-rbac-proxy/0.log" Jul 16 22:20:24.650292 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.650262 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_042787c0-eece-41ee-b1ad-12bcd75922f4/kube-rbac-proxy-metric/0.log" Jul 16 22:20:24.675502 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.675475 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_042787c0-eece-41ee-b1ad-12bcd75922f4/prom-label-proxy/0.log" Jul 16 22:20:24.699125 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.699098 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_042787c0-eece-41ee-b1ad-12bcd75922f4/init-config-reloader/0.log" Jul 16 22:20:24.783422 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.783392 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-hqjsh_362547e8-d75f-4861-b842-d5aec9f80ba9/kube-state-metrics/0.log" Jul 16 22:20:24.805997 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.805963 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-hqjsh_362547e8-d75f-4861-b842-d5aec9f80ba9/kube-rbac-proxy-main/0.log" Jul 16 22:20:24.830094 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.830061 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-hqjsh_362547e8-d75f-4861-b842-d5aec9f80ba9/kube-rbac-proxy-self/0.log" Jul 16 22:20:24.864313 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.864224 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bbd464946-dpq6k_69485f8e-111b-4abe-8669-549bb5bb6fe5/metrics-server/0.log" Jul 16 22:20:24.892978 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:24.892942 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-c6jq8_0ecf63b0-0e83-46ad-ac13-548c487771b1/monitoring-plugin/0.log" Jul 16 22:20:25.029706 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.029678 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d9jbv_d6e14b29-ad65-44ed-8119-8333506ad460/node-exporter/0.log" Jul 16 22:20:25.051319 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.051281 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d9jbv_d6e14b29-ad65-44ed-8119-8333506ad460/kube-rbac-proxy/0.log" Jul 16 22:20:25.073465 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.073438 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d9jbv_d6e14b29-ad65-44ed-8119-8333506ad460/init-textfile/0.log" Jul 16 22:20:25.188725 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.188639 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-q8jcj_f9b83b31-e9fa-40eb-854d-05d789cb77cd/kube-rbac-proxy-main/0.log" Jul 16 22:20:25.211756 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.211733 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-q8jcj_f9b83b31-e9fa-40eb-854d-05d789cb77cd/kube-rbac-proxy-self/0.log" Jul 16 22:20:25.236503 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.236469 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-q8jcj_f9b83b31-e9fa-40eb-854d-05d789cb77cd/openshift-state-metrics/0.log" Jul 16 22:20:25.268650 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.268618 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_87572c86-25d0-48aa-89a2-8d4fe36e616b/prometheus/0.log" Jul 16 22:20:25.294341 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.294291 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_87572c86-25d0-48aa-89a2-8d4fe36e616b/config-reloader/0.log" Jul 16 22:20:25.316680 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.316652 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_87572c86-25d0-48aa-89a2-8d4fe36e616b/thanos-sidecar/0.log" Jul 16 22:20:25.337618 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.337589 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_87572c86-25d0-48aa-89a2-8d4fe36e616b/kube-rbac-proxy-web/0.log" Jul 16 22:20:25.360050 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.360016 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_87572c86-25d0-48aa-89a2-8d4fe36e616b/kube-rbac-proxy/0.log" Jul 16 22:20:25.383068 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.383037 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_87572c86-25d0-48aa-89a2-8d4fe36e616b/kube-rbac-proxy-thanos/0.log" Jul 16 22:20:25.406669 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.406643 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_87572c86-25d0-48aa-89a2-8d4fe36e616b/init-config-reloader/0.log" Jul 16 22:20:25.440814 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.440726 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-rr5fg_c17c3042-2778-4bd3-a6e4-48bd3910f356/prometheus-operator/0.log" Jul 16 22:20:25.466439 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.466408 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-rr5fg_c17c3042-2778-4bd3-a6e4-48bd3910f356/kube-rbac-proxy/0.log" Jul 16 22:20:25.492646 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.492615 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-xxqr6_2eabe3d7-5ebe-43b4-98b7-8ca39a1c6c2a/prometheus-operator-admission-webhook/0.log" Jul 16 22:20:25.526574 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.526538 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75cccb9b5b-6cjgv_816ddd8c-e681-4b27-84f3-7ad1881838f8/telemeter-client/0.log" Jul 16 22:20:25.550591 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.550552 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75cccb9b5b-6cjgv_816ddd8c-e681-4b27-84f3-7ad1881838f8/reload/0.log" Jul 16 22:20:25.576336 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.576303 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75cccb9b5b-6cjgv_816ddd8c-e681-4b27-84f3-7ad1881838f8/kube-rbac-proxy/0.log" Jul 16 22:20:25.608467 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.608436 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bd44f97fc-mvdsv_1357ede3-e5c3-403f-806f-258928451826/thanos-query/0.log" Jul 16 22:20:25.632894 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.632864 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bd44f97fc-mvdsv_1357ede3-e5c3-403f-806f-258928451826/kube-rbac-proxy-web/0.log" Jul 16 22:20:25.659894 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.659861 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bd44f97fc-mvdsv_1357ede3-e5c3-403f-806f-258928451826/kube-rbac-proxy/0.log" Jul 16 22:20:25.682932 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.682891 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bd44f97fc-mvdsv_1357ede3-e5c3-403f-806f-258928451826/prom-label-proxy/0.log" Jul 16 22:20:25.706106 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.706015 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bd44f97fc-mvdsv_1357ede3-e5c3-403f-806f-258928451826/kube-rbac-proxy-rules/0.log" Jul 16 22:20:25.730829 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:25.730793 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bd44f97fc-mvdsv_1357ede3-e5c3-403f-806f-258928451826/kube-rbac-proxy-metrics/0.log" Jul 16 22:20:26.849174 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.849130 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6"] Jul 16 22:20:26.849753 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.849732 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9745b6-5195-4fdb-8408-9a490d28515c" containerName="test-curl-7573-140428994112368" Jul 16 22:20:26.849843 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.849757 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9745b6-5195-4fdb-8408-9a490d28515c" containerName="test-curl-7573-140428994112368" Jul 16 22:20:26.849902 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.849860 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9745b6-5195-4fdb-8408-9a490d28515c" containerName="test-curl-7573-140428994112368" Jul 16 22:20:26.853131 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.853105 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:26.855615 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.855580 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qvtkv\"/\"kube-root-ca.crt\"" Jul 16 22:20:26.855788 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.855648 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qvtkv\"/\"openshift-service-ca.crt\"" Jul 16 22:20:26.856635 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.856614 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qvtkv\"/\"default-dockercfg-bx2pd\"" Jul 16 22:20:26.863141 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.863111 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6"] Jul 16 22:20:26.996530 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.996475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-sys\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:26.996746 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.996541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-proc\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:26.996746 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.996638 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-podres\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:26.996862 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.996787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-lib-modules\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:26.996862 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:26.996829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9svq8\" (UniqueName: \"kubernetes.io/projected/8c935706-6a9e-417b-a1e9-686d3ba46bb7-kube-api-access-9svq8\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097447 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-podres\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097632 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-lib-modules\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097632 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9svq8\" (UniqueName: \"kubernetes.io/projected/8c935706-6a9e-417b-a1e9-686d3ba46bb7-kube-api-access-9svq8\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097724 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097571 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-podres\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097724 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-sys\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097724 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097643 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-lib-modules\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097724 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-proc\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097869 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-proc\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.097869 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.097761 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c935706-6a9e-417b-a1e9-686d3ba46bb7-sys\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.106087 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.106009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9svq8\" (UniqueName: \"kubernetes.io/projected/8c935706-6a9e-417b-a1e9-686d3ba46bb7-kube-api-access-9svq8\") pod \"perf-node-gather-daemonset-d4ht6\" (UID: \"8c935706-6a9e-417b-a1e9-686d3ba46bb7\") " pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.164237 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.164187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:27.317439 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:27.317410 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6"] Jul 16 22:20:27.319537 ip-10-0-133-93 kubenswrapper[2573]: W0716 22:20:27.319495 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c935706_6a9e_417b_a1e9_686d3ba46bb7.slice/crio-270419a99360488eedd1874e7a7b08317cb90b5c89293afaf0f01ceebd75df82 WatchSource:0}: Error finding container 270419a99360488eedd1874e7a7b08317cb90b5c89293afaf0f01ceebd75df82: Status 404 returned error can't find the container with id 270419a99360488eedd1874e7a7b08317cb90b5c89293afaf0f01ceebd75df82 Jul 16 22:20:28.127803 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:28.127774 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6456f767bd-zpnmp_09d0e255-5d9b-4928-be38-1c5331e8fc11/console/0.log" Jul 16 22:20:28.140454 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:28.140422 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" event={"ID":"8c935706-6a9e-417b-a1e9-686d3ba46bb7","Type":"ContainerStarted","Data":"bdce9a666e1d615706e423c50559b1f56c94cceee999897adf2b0c2a98e8b372"} Jul 16 22:20:28.140682 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:28.140480 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" event={"ID":"8c935706-6a9e-417b-a1e9-686d3ba46bb7","Type":"ContainerStarted","Data":"270419a99360488eedd1874e7a7b08317cb90b5c89293afaf0f01ceebd75df82"} Jul 16 22:20:28.140682 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:28.140503 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:28.157962 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:28.157909 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" podStartSLOduration=2.157890975 podStartE2EDuration="2.157890975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:28.155728349 +0000 UTC m=+2475.197952588" watchObservedRunningTime="2026-07-16 22:20:28.157890975 +0000 UTC m=+2475.200115213" Jul 16 22:20:29.861968 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:29.861938 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mwxc9_1e1f0037-10ff-44e8-9b40-7979eace45a5/dns/0.log" Jul 16 22:20:29.882986 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:29.882959 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mwxc9_1e1f0037-10ff-44e8-9b40-7979eace45a5/kube-rbac-proxy/0.log" Jul 16 22:20:29.911020 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:29.910985 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jktm_b11b603d-097d-4bca-b187-43a143774e00/dns-node-resolver/0.log" Jul 16 22:20:30.545948 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:30.545923 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knpg7_c91f3cb3-36af-4a18-8e17-0da940e6ef85/node-ca/0.log" Jul 16 22:20:31.755137 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:31.755108 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b5c5c797-ztsd5_681b65a0-b05d-43c0-8ebf-8e1483b2bd26/kube-auth-proxy/0.log" Jul 16 22:20:31.908140 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:31.908106 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vmmsf_a5e5398f-c17b-405e-8855-67a8fdbb929c/payload-processing/0.log" Jul 16 22:20:32.630901 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:32.630873 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgf2x_d86a9fa6-dda8-43f2-9330-fa89e82d0e61/serve-healthcheck-canary/0.log" Jul 16 22:20:33.223818 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:33.223788 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdc8v_b1872486-e142-4f26-8afb-aed610a1db52/kube-rbac-proxy/0.log" Jul 16 22:20:33.246870 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:33.246842 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdc8v_b1872486-e142-4f26-8afb-aed610a1db52/exporter/0.log" Jul 16 22:20:33.269612 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:33.269582 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdc8v_b1872486-e142-4f26-8afb-aed610a1db52/extractor/0.log" Jul 16 22:20:34.156439 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:34.156404 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qvtkv/perf-node-gather-daemonset-d4ht6" Jul 16 22:20:36.000155 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:36.000051 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-db5748bcb-6vwzm_67a5213d-5613-4946-a352-8ae8f2443346/manager/0.log" Jul 16 22:20:36.048795 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:36.048764 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-db5748bcb-6vwzm_67a5213d-5613-4946-a352-8ae8f2443346/copy-manifests/0.log" Jul 16 22:20:37.454160 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:37.454129 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b5c5c797-ztsd5_681b65a0-b05d-43c0-8ebf-8e1483b2bd26/kube-auth-proxy/0.log" Jul 16 22:20:37.493283 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:37.493256 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6844f9fcbb-2drxh_ac89078a-70a8-4849-a0b1-5a18840bf108/manager/0.log" Jul 16 22:20:37.569769 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:37.569735 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vmmsf_a5e5398f-c17b-405e-8855-67a8fdbb929c/payload-processing/0.log" Jul 16 22:20:39.139883 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:39.139857 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b5c5c797-ztsd5_681b65a0-b05d-43c0-8ebf-8e1483b2bd26/kube-auth-proxy/0.log" Jul 16 22:20:39.268096 ip-10-0-133-93 kubenswrapper[2573]: I0716 22:20:39.268057 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vmmsf_a5e5398f-c17b-405e-8855-67a8fdbb929c/payload-processing/0.log"