Jul 23 11:26:47.868957 ip-10-0-129-204 systemd[1]: Starting Kubernetes Kubelet... Jul 23 11:26:48.244007 ip-10-0-129-204 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:26:48.244007 ip-10-0-129-204 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 11:26:48.244007 ip-10-0-129-204 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:26:48.244007 ip-10-0-129-204 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 11:26:48.244007 ip-10-0-129-204 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:26:48.246703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.246599 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 11:26:48.250722 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250694 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:48.250722 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250721 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:48.250722 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250725 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:48.250722 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250728 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250731 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250735 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250737 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250743 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250747 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250750 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250753 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250756 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250759 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250762 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250765 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250768 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250771 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250774 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250777 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250781 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250784 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250787 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:48.250858 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250789 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250792 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250795 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250797 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250800 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250802 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250805 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250808 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250811 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250814 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250816 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250819 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250821 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250824 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250826 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250830 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250833 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250836 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250839 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250842 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:48.251329 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250844 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250846 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250850 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250852 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250860 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250863 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250866 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250869 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250871 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250874 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250877 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250880 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250883 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250885 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250888 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250890 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250892 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250895 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250897 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250901 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:48.251847 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250903 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250905 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250908 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250910 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250912 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250915 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250917 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250920 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250923 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250926 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250928 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250931 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250933 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250937 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250941 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250944 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250946 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250948 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250951 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250953 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:48.252342 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250956 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250958 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250961 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250966 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250970 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250974 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250976 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250979 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250981 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250984 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250986 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250988 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250991 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250993 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250996 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.250998 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251000 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251797 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251806 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251809 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:48.252864 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251812 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251815 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251818 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251821 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251823 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251826 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251828 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251831 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251836 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251839 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251842 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251845 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251848 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251851 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251854 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251857 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251861 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251864 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251867 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251870 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:48.253339 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251872 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251875 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251878 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251880 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251883 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251886 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251888 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251891 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251893 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251896 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251898 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251902 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251905 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251907 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251910 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251913 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251915 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251918 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251920 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251927 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:48.253896 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251930 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251932 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251934 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251937 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251939 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251942 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251966 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251969 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251972 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251978 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251980 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251985 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251988 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251992 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251996 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.251999 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252002 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252005 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252008 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:48.254394 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252011 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252014 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252017 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252019 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252023 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252028 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252031 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252034 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252037 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252039 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252042 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252044 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252047 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252049 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252052 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252054 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252057 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252059 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252062 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:48.254879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252064 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252066 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252069 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252071 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252079 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252082 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252084 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252087 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252089 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252092 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252094 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252096 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252099 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252101 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252104 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252106 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252109 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252112 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252508 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252513 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:48.255352 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252516 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252519 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252522 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252527 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252531 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252534 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252537 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252540 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252542 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252545 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252547 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252550 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252552 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252554 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252557 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252559 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252562 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252582 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252585 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:48.255859 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252588 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252590 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252593 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252596 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252598 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252601 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252603 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252605 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252608 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252611 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252614 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252617 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252620 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252622 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252625 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252628 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252631 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252633 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252636 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:48.256325 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252638 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252641 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252644 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252647 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252649 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252652 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252654 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252657 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252660 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252662 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252665 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252669 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252671 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252674 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252676 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252679 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252682 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252684 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252687 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252689 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:48.256830 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252692 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252694 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252697 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252700 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252702 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252705 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252707 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252709 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252712 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252714 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252718 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252720 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252723 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252725 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252727 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252730 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252733 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252735 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252737 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:48.257315 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252740 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252743 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252747 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252749 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252752 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252755 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252758 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252760 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252764 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252767 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252769 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252772 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252774 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252777 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252779 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252782 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252785 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252787 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252791 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.252794 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:48.257915 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252876 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252889 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252895 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252900 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252904 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252908 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252912 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252916 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252920 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252923 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252926 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252929 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252933 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252936 2556 flags.go:64] FLAG: --cgroup-root="" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252939 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252942 2556 flags.go:64] FLAG: --client-ca-file="" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252944 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252947 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252952 2556 flags.go:64] FLAG: --cluster-domain="" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252955 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252958 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252962 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252965 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252969 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 11:26:48.258409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252972 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252975 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252978 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252981 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252984 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252987 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252990 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252993 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.252997 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253000 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253002 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253005 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253008 2556 flags.go:64] FLAG: --enable-server="true" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253011 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253015 2556 flags.go:64] FLAG: --event-burst="100" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253018 2556 flags.go:64] FLAG: --event-qps="50" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253020 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253023 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253026 2556 flags.go:64] FLAG: --eviction-hard="" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253030 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253033 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253036 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253039 2556 flags.go:64] FLAG: --eviction-soft="" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253042 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253045 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 11:26:48.259045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253048 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253051 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253054 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253057 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253060 2556 flags.go:64] FLAG: --feature-gates="" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253064 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253067 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253070 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253073 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253076 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253079 2556 flags.go:64] FLAG: --help="false" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253082 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253084 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253087 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253090 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253094 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253097 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253100 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253102 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253105 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253108 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253111 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253114 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253116 2556 flags.go:64] FLAG: --kube-reserved="" Jul 23 11:26:48.259703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253119 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253122 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253125 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253127 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253130 2556 flags.go:64] FLAG: --lock-file="" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253133 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253136 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253139 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253147 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253150 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253153 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253156 2556 flags.go:64] FLAG: --logging-format="text" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253159 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253162 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253165 2556 flags.go:64] FLAG: --manifest-url="" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253168 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253173 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253176 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253180 2556 flags.go:64] FLAG: --max-pods="110" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253183 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253185 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253188 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253191 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253194 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253197 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 11:26:48.260334 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253200 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253208 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253211 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253214 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253217 2556 flags.go:64] FLAG: --pod-cidr="" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253223 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253230 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253233 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253236 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253240 2556 flags.go:64] FLAG: --port="10250" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253243 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253246 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cc4487a8dce76742" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253249 2556 flags.go:64] FLAG: --qos-reserved="" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253252 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253255 2556 flags.go:64] FLAG: --register-node="true" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253258 2556 flags.go:64] FLAG: --register-with-taints="" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253262 2556 flags.go:64] FLAG: --registry-burst="10" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253265 2556 flags.go:64] FLAG: --registry-qps="5" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253268 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253271 2556 flags.go:64] FLAG: --reserved-memory="" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253275 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253278 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253281 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253284 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253287 2556 flags.go:64] FLAG: --runonce="false" Jul 23 11:26:48.260985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253290 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253292 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253295 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253298 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253301 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253304 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253307 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253310 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253313 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253316 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253318 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253321 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253325 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253328 2556 flags.go:64] FLAG: --system-cgroups="" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253331 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253336 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253339 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253342 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253346 2556 flags.go:64] FLAG: --tls-min-version="" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253349 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253352 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253355 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253358 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253361 2556 flags.go:64] FLAG: --v="2" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253366 2556 flags.go:64] FLAG: --version="false" Jul 23 11:26:48.261941 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253371 2556 flags.go:64] FLAG: --vmodule="" Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253375 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.253379 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253500 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253504 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253507 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253510 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253513 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253516 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253519 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253521 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253524 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253526 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253529 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253532 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253534 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253537 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253539 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253542 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253546 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:48.262746 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253549 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253551 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253554 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253557 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253559 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253562 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253581 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253584 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253587 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253589 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253592 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253596 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253599 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253601 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253604 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253608 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253611 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253614 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253616 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:48.263273 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253619 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253621 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253624 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253626 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253628 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253631 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253633 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253637 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253640 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253644 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253646 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253649 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253653 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253656 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253659 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253663 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253666 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253670 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253673 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:48.263757 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253676 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253679 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253681 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253684 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253687 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253691 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253693 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253696 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253699 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253701 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253704 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253707 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253709 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253712 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253715 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253717 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253720 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253723 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253725 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253728 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:48.264220 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253730 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253733 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253735 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253738 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253740 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253744 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253747 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253749 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253752 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253754 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253756 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253759 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253762 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253764 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253767 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253769 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253772 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253776 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253778 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253781 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:48.264726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253784 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253786 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253788 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.253791 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.254450 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.263374 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.263398 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.263420 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.263431 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263497 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263503 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263507 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263510 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263513 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263516 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:48.265222 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263519 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263522 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263525 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263527 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263530 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263532 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263535 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263538 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263543 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263547 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263551 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263554 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263557 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263559 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263562 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263564 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263582 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263585 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263587 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:48.265642 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263590 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263592 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263595 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263598 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263602 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263605 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263608 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263610 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263613 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263616 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263618 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263621 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263624 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263627 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263629 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263633 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263636 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263639 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263642 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263645 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:48.266099 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263648 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263651 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263654 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263657 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263660 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263662 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263665 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263667 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263671 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263674 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263676 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263679 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263681 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263684 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263687 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263690 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263693 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263696 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263699 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263701 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:48.266698 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263704 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263707 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263709 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263712 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263714 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263717 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263719 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263721 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263724 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263726 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263729 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263731 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263734 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263736 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263740 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263742 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263745 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263747 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263750 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263752 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:48.267208 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263756 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263759 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263762 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263764 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263767 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263771 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263775 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263778 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263781 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263783 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263786 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263789 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263791 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263794 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:48.267726 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.263800 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263930 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263936 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263939 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263941 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263944 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263947 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263949 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263952 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263954 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263957 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263959 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263962 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263965 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263967 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263969 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263972 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263974 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263977 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263980 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:48.268085 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263983 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263985 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263988 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263990 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263993 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263995 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.263998 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264000 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264003 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264006 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264008 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264011 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264013 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264016 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264018 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264021 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264023 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264026 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264029 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:48.268580 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264031 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264034 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264036 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264038 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264041 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264043 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264046 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264048 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264051 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264053 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264056 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264059 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264061 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264064 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264067 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264070 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264073 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264075 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264078 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264080 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:48.269053 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264083 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264086 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264089 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264092 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264094 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264097 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264099 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264102 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264104 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264106 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264109 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264113 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264116 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264119 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264122 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264125 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264128 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264130 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264133 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:48.269550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264136 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264140 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264143 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264146 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264149 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264152 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264154 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264157 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264159 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264162 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264165 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264167 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264170 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264172 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264175 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264177 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264180 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264182 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264185 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264187 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:48.270042 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264190 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:48.270535 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:48.264193 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:48.270535 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.264198 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:26:48.270535 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.264383 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 11:26:48.270535 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.267446 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 11:26:48.270535 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.268520 2556 server.go:1019] "Starting client certificate rotation" Jul 23 11:26:48.270535 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.268630 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 11:26:48.270535 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.268668 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 11:26:48.271053 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.271032 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 11:26:48.272406 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.272387 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 11:26:48.287712 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.287681 2556 log.go:25] "Validated CRI v1 runtime API" Jul 23 11:26:48.289943 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.289925 2556 log.go:25] "Validated CRI v1 image API" Jul 23 11:26:48.291664 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.291639 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 11:26:48.294302 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.294267 2556 fs.go:135] Filesystem UUIDs: map[648bc398-afe5-4941-930c-4e34e0792f51:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d88a9d2f-ca30-4181-90ac-9ef1f3ae5640:/dev/nvme0n1p4] Jul 23 11:26:48.294401 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.294299 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 11:26:48.297174 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.297137 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 11:26:48.300489 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.300351 2556 manager.go:216] Machine: {Timestamp:2026-07-23 11:26:48.299791955 +0000 UTC m=+0.317678581 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100495 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea49a46cb81833224017733eeeec9 SystemUUID:ec2ea49a-46cb-8183-3224-017733eeeec9 BootID:0bfbf112-cda8-49b9-80f7-6091d31f3977 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f6:2d:38:28:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f6:2d:38:28:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:83:60:e7:dd:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 11:26:48.300489 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.300467 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 11:26:48.300688 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.300583 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 11:26:48.302846 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.302805 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 11:26:48.302996 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.302848 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 11:26:48.303548 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.303535 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 11:26:48.303614 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.303552 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 11:26:48.303614 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.303560 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 11:26:48.304808 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.304793 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 11:26:48.305830 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.305816 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 11:26:48.308585 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.308553 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 23 11:26:48.308749 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.308737 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 11:26:48.308795 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.308773 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 23 11:26:48.308795 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.308784 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 11:26:48.308868 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.308815 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 11:26:48.308868 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.308827 2556 kubelet.go:398] "Adding apiserver pod source" Jul 23 11:26:48.308868 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.308836 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 11:26:48.309273 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.309258 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 11:26:48.309339 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.309292 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 11:26:48.312107 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.312083 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 11:26:48.313748 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.313731 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 11:26:48.313799 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.313766 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 11:26:48.315044 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315021 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 11:26:48.315044 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315038 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 11:26:48.315044 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315044 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315051 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315060 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315065 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315073 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315090 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315098 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315107 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315117 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315123 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315151 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 11:26:48.315257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.315156 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 11:26:48.317636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.317609 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ktjgh" Jul 23 11:26:48.318824 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.318809 2556 server.go:1268] "Started kubelet" Jul 23 11:26:48.318951 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.318899 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 11:26:48.319089 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.319058 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 11:26:48.319591 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.319544 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-204.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 11:26:48.319707 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.319559 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 11:26:48.319769 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.319707 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 11:26:48.319867 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.319838 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 11:26:48.320008 ip-10-0-129-204 systemd[1]: Started Kubernetes Kubelet. Jul 23 11:26:48.320878 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.320838 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 11:26:48.322442 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.322422 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 23 11:26:48.324448 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.324427 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 11:26:48.324564 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.323704 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-204.ec2.internal.18c4e79bb6d2c7df default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-204.ec2.internal,UID:ip-10-0-129-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-204.ec2.internal,},FirstTimestamp:2026-07-23 11:26:48.318773215 +0000 UTC m=+0.336659840,LastTimestamp:2026-07-23 11:26:48.318773215 +0000 UTC m=+0.336659840,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-204.ec2.internal,}" Jul 23 11:26:48.324996 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.324976 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ktjgh" Jul 23 11:26:48.326912 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.326896 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 11:26:48.327589 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.327551 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 11:26:48.327728 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.327709 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:48.328218 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.328199 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 11:26:48.328306 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.328221 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 11:26:48.328531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.328502 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 11:26:48.328817 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.328763 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 23 11:26:48.328817 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.328774 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 23 11:26:48.329074 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.329017 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 11:26:48.329074 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.329036 2556 factory.go:55] Registering systemd factory Jul 23 11:26:48.329074 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.329043 2556 factory.go:223] Registration of the systemd container factory successfully Jul 23 11:26:48.329310 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.329280 2556 factory.go:153] Registering CRI-O factory Jul 23 11:26:48.329310 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.329293 2556 factory.go:223] Registration of the crio container factory successfully Jul 23 11:26:48.329408 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.329315 2556 factory.go:103] Registering Raw factory Jul 23 11:26:48.329408 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.329333 2556 manager.go:1196] Started watching for new ooms in manager Jul 23 11:26:48.329884 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.329864 2556 manager.go:349] Starting recovery of all containers Jul 23 11:26:48.330193 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.330152 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 11:26:48.330289 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.330263 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 11:26:48.337135 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.337071 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 11:26:48.343848 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.343701 2556 manager.go:354] Recovery completed Jul 23 11:26:48.348228 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.348208 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:48.350622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.350563 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:48.350692 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.350641 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:48.350692 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.350656 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:48.351319 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.351294 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 11:26:48.351319 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.351318 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 11:26:48.351408 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.351304 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:48.351408 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.351335 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 23 11:26:48.352067 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.352050 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:48.352126 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.352082 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:48.352126 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.352093 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:48.352527 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.352509 2556 policy_none.go:49] "None policy: Start" Jul 23 11:26:48.352527 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.352530 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 11:26:48.352674 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.352541 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 11:26:48.353599 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.353586 2556 policy_none.go:47] "Start" Jul 23 11:26:48.391324 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.391299 2556 manager.go:337] "Starting Device Plugin manager" Jul 23 11:26:48.391486 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.391362 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 11:26:48.391486 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.391375 2556 server.go:87] "Starting device plugin registration server" Jul 23 11:26:48.391716 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.391702 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 11:26:48.391799 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.391720 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 11:26:48.391858 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.391836 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 11:26:48.391956 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.391934 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 11:26:48.391956 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.391950 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 11:26:48.392534 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.392512 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 11:26:48.392644 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.392593 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:48.491842 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.491810 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:48.492867 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.492853 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:48.492936 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.492889 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:48.492936 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.492901 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:48.492936 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.492928 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.493616 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.493598 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 11:26:48.493707 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.493645 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 11:26:48.493707 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.493672 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 11:26:48.493805 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.493717 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 11:26:48.495590 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.495516 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:48.501647 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.501621 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.501647 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.501653 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-204.ec2.internal\": node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:48.574099 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.574071 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:48.594209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.594177 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal"] Jul 23 11:26:48.594294 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.594255 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:48.595246 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.595226 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:48.595323 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.595263 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:48.595323 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.595276 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:48.596528 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.596511 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:48.596675 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.596658 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.596736 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.596698 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:48.597355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.597337 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:48.597438 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.597371 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:48.597438 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.597381 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:48.597438 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.597343 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:48.597545 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.597443 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:48.597545 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.597455 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:48.598536 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.598522 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.598607 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.598555 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:48.599397 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.599369 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:48.599397 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.599397 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:48.599536 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.599409 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:48.615282 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.615258 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-204.ec2.internal\" not found" node="ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.619249 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.619228 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-204.ec2.internal\" not found" node="ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.674374 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.674331 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:48.730351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.730310 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/202d9bf1b3a2ad64be78b9941f522638-config\") pod \"kube-apiserver-proxy-ip-10-0-129-204.ec2.internal\" (UID: \"202d9bf1b3a2ad64be78b9941f522638\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.730351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.730352 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34fc63d6b59e151c5bd785e99c9b32e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"34fc63d6b59e151c5bd785e99c9b32e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.730557 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.730373 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34fc63d6b59e151c5bd785e99c9b32e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"34fc63d6b59e151c5bd785e99c9b32e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.774937 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.774860 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:48.831311 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.831279 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34fc63d6b59e151c5bd785e99c9b32e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"34fc63d6b59e151c5bd785e99c9b32e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.831311 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.831309 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34fc63d6b59e151c5bd785e99c9b32e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"34fc63d6b59e151c5bd785e99c9b32e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.831461 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.831330 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/202d9bf1b3a2ad64be78b9941f522638-config\") pod \"kube-apiserver-proxy-ip-10-0-129-204.ec2.internal\" (UID: \"202d9bf1b3a2ad64be78b9941f522638\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.831461 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.831366 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34fc63d6b59e151c5bd785e99c9b32e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"34fc63d6b59e151c5bd785e99c9b32e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.831461 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.831383 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/202d9bf1b3a2ad64be78b9941f522638-config\") pod \"kube-apiserver-proxy-ip-10-0-129-204.ec2.internal\" (UID: \"202d9bf1b3a2ad64be78b9941f522638\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.831461 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.831392 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34fc63d6b59e151c5bd785e99c9b32e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"34fc63d6b59e151c5bd785e99c9b32e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.875673 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.875624 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:48.920207 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.920173 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.921377 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:48.921340 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" Jul 23 11:26:48.976840 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:48.976790 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:49.078793 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.078681 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found" Jul 23 11:26:49.144795 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.144769 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:49.230930 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.230887 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" Jul 23 11:26:49.248174 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.248146 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 11:26:49.250145 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.250122 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" Jul 23 11:26:49.260331 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.260310 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 11:26:49.268021 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.267980 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 11:26:49.268189 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.268163 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 11:26:49.268601 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.268170 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 11:26:49.310405 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.310362 2556 apiserver.go:52] "Watching apiserver" Jul 23 11:26:49.327672 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.327642 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 11:26:49.328995 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.328873 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 11:21:49 +0000 UTC" deadline="2028-03-24 14:00:52.593904163 +0000 UTC" Jul 23 11:26:49.328995 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.328939 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14642h34m3.264968406s" Jul 23 11:26:49.330321 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.330299 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 11:26:49.330742 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.330718 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-r5v5q","openshift-multus/multus-additional-cni-plugins-n952k","openshift-multus/multus-b52vr","openshift-network-diagnostics/network-check-target-tn5xd","openshift-ovn-kubernetes/ovnkube-node-mnjnv","kube-system/global-pull-secret-syncer-4bqk7","kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs","openshift-cluster-node-tuning-operator/tuned-zgxd4","openshift-dns/node-resolver-bxgc5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal","openshift-multus/network-metrics-daemon-zgbls","openshift-network-operator/iptables-alerter-v9hgw","kube-system/konnectivity-agent-gznv2"] Jul 23 11:26:49.332383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.332364 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.333701 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.333681 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.334850 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.334825 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 11:26:49.334850 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.334837 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 11:26:49.335036 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.334827 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 11:26:49.335036 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.334975 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q7886\"" Jul 23 11:26:49.335328 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.335313 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.335739 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.335719 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 11:26:49.336167 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.336145 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 11:26:49.336314 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.336145 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 11:26:49.336314 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.336151 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 11:26:49.336453 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.336438 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vk9q9\"" Jul 23 11:26:49.336520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.336504 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 11:26:49.336569 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.336506 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:49.336656 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.336629 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:26:49.337517 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.337502 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 11:26:49.337829 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.337809 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kl9lq\"" Jul 23 11:26:49.337985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.337969 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.339438 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.339417 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.339542 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.339479 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:26:49.340257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.340239 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 11:26:49.340462 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.340444 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 11:26:49.340534 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.340468 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 11:26:49.340534 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.340481 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 11:26:49.340534 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.340445 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 11:26:49.340749 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.340735 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 11:26:49.340969 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.340953 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.341047 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.340986 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9pd9p\"" Jul 23 11:26:49.341778 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.341759 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 11:26:49.342414 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.342400 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.343939 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.343924 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.345192 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.345168 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:26:49.345284 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.345214 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 11:26:49.345355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.345174 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 11:26:49.345486 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.345470 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 11:26:49.345544 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.345474 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 11:26:49.345618 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.345595 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:49.345676 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.345618 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkvkg\"" Jul 23 11:26:49.345676 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.345643 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d2gdp\"" Jul 23 11:26:49.345676 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.345661 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:26:49.346318 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.346300 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 11:26:49.346386 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.346371 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 11:26:49.346469 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.346452 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9g76b\"" Jul 23 11:26:49.347146 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.347127 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.348720 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.348701 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:26:49.353905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.351159 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 11:26:49.353905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.351239 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 11:26:49.353905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.351536 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 11:26:49.353905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.351619 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r6bn9\"" Jul 23 11:26:49.353905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.351758 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:26:49.353905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.352020 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 11:26:49.354242 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.351966 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7knnw\"" Jul 23 11:26:49.366811 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.366722 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vs2qf" Jul 23 11:26:49.375040 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.375009 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vs2qf" Jul 23 11:26:49.429874 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.429845 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 11:26:49.434890 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.434863 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-host\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.434976 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.434899 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-kubelet\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.434976 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.434917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-hostroot\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.434976 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.434951 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.434976 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.434974 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-registration-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.435129 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.434993 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58ct\" (UniqueName: \"kubernetes.io/projected/1e5e40a3-78e5-4616-9d69-45ae928aa773-kube-api-access-p58ct\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.435129 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435009 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-log-socket\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435129 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435025 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435129 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435067 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovnkube-script-lib\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435129 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435099 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-run\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.435129 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435118 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-sys\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.435338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435143 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-lib-modules\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.435338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-conf-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.435338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435215 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9n2\" (UniqueName: \"kubernetes.io/projected/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-kube-api-access-pm9n2\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.435338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435232 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19c37710-4dfc-4e47-82dc-3622c2ace92c-serviceca\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.435338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435248 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-var-lib-kubelet\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.435338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-sys-fs\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.435338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435296 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e5e40a3-78e5-4616-9d69-45ae928aa773-host-slash\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.435338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435325 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e080c8a4-51cf-4821-bade-e8378e6690ce-konnectivity-ca\") pod \"konnectivity-agent-gznv2\" (UID: \"e080c8a4-51cf-4821-bade-e8378e6690ce\") " pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435347 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-ovn\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435381 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovnkube-config\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435399 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435418 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-var-lib-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435436 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-modprobe-d\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435469 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-systemd\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435485 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-socket-dir-parent\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435501 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435519 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435550 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435567 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-kubelet\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.435622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435581 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-slash\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435624 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-cni-netd\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435663 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-env-overrides\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435684 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9dde853-d0e2-40da-bb68-ab7f67f456a7-tmp-dir\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435706 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435723 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57d8172a-a0cd-4627-9e29-a03caae5da3c-tmp\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435738 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn52b\" (UniqueName: \"kubernetes.io/projected/c9dde853-d0e2-40da-bb68-ab7f67f456a7-kube-api-access-wn52b\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435775 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-cnibin\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435795 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-k8s-cni-cncf-io\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435811 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-device-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435826 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e080c8a4-51cf-4821-bade-e8378e6690ce-agent-certs\") pod \"konnectivity-agent-gznv2\" (UID: \"e080c8a4-51cf-4821-bade-e8378e6690ce\") " pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435855 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovn-node-metrics-cert\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435880 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysconfig\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdc8\" (UniqueName: \"kubernetes.io/projected/57d8172a-a0cd-4627-9e29-a03caae5da3c-kube-api-access-zhdc8\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435920 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-cni-bin\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435937 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-cni-bin\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.436038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435976 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-etc-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.435993 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-cni-multus\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/540f8401-c269-48d4-8475-00950f61fd03-multus-daemon-config\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436026 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brlk9\" (UniqueName: \"kubernetes.io/projected/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-kube-api-access-brlk9\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436042 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-system-cni-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-systemd\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2wk\" (UniqueName: \"kubernetes.io/projected/2b616b9b-a4ac-4feb-a082-6eae00565d60-kube-api-access-ph2wk\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436096 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/540f8401-c269-48d4-8475-00950f61fd03-cni-binary-copy\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436113 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436146 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-multus-certs\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436168 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/110693d2-151e-459c-8789-094e101446fc-dbus\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436184 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-socket-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436215 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436232 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19c37710-4dfc-4e47-82dc-3622c2ace92c-host\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436250 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-system-cni-dir\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436271 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-cni-binary-copy\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.436512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436295 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-os-release\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436311 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-etc-kubernetes\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436327 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/110693d2-151e-459c-8789-094e101446fc-kubelet-config\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436346 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9dde853-d0e2-40da-bb68-ab7f67f456a7-hosts-file\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436366 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2xd\" (UniqueName: \"kubernetes.io/projected/19c37710-4dfc-4e47-82dc-3622c2ace92c-kube-api-access-gr2xd\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436382 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5p7x\" (UniqueName: \"kubernetes.io/projected/15efe522-b59d-40f5-982b-00b752ea3fa6-kube-api-access-n5p7x\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436399 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-run-netns\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436414 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysctl-d\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436430 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwqb\" (UniqueName: \"kubernetes.io/projected/540f8401-c269-48d4-8475-00950f61fd03-kube-api-access-tpwqb\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436455 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-node-log\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-cnibin\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436516 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-systemd-units\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436534 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-cni-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436550 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-netns\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436582 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e5e40a3-78e5-4616-9d69-45ae928aa773-iptables-alerter-script\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.437003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436604 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-kubernetes\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.437533 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436624 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-os-release\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.437533 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436651 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysctl-conf\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.437533 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.436680 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-tuned\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.451608 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.451582 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:49.538036 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.537990 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-systemd-units\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.538036 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538037 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-cni-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538067 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-netns\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538090 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e5e40a3-78e5-4616-9d69-45ae928aa773-iptables-alerter-script\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538114 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-kubernetes\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-systemd-units\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538159 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-cni-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-netns\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538194 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-kubernetes\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-os-release\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538255 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysctl-conf\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.538278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-os-release\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538286 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-tuned\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538310 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-host\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538333 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-kubelet\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538354 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-hostroot\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538373 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-host\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538397 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-kubelet\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538404 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysctl-conf\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538415 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538425 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-registration-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538437 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-hostroot\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538455 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58ct\" (UniqueName: \"kubernetes.io/projected/1e5e40a3-78e5-4616-9d69-45ae928aa773-kube-api-access-p58ct\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-log-socket\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538503 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538520 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovnkube-script-lib\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538535 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-registration-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538574 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 11:26:49.538752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538545 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-log-socket\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538539 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-run\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538739 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-sys\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538756 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-lib-modules\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538774 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-conf-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538795 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9n2\" (UniqueName: \"kubernetes.io/projected/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-kube-api-access-pm9n2\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538600 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-run\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538820 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19c37710-4dfc-4e47-82dc-3622c2ace92c-serviceca\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538827 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-sys\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538592 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538875 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-conf-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538915 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-var-lib-kubelet\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538945 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-sys-fs\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538973 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e5e40a3-78e5-4616-9d69-45ae928aa773-host-slash\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.538960 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-lib-modules\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539006 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e080c8a4-51cf-4821-bade-e8378e6690ce-konnectivity-ca\") pod \"konnectivity-agent-gznv2\" (UID: \"e080c8a4-51cf-4821-bade-e8378e6690ce\") " pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539033 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-ovn\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539045 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-sys-fs\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.539636 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovnkube-config\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539086 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539102 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-var-lib-kubelet\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-var-lib-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539145 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e5e40a3-78e5-4616-9d69-45ae928aa773-host-slash\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539173 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-modprobe-d\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539181 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovnkube-script-lib\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-systemd\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539226 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-socket-dir-parent\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539229 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-ovn\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539254 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539310 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-var-lib-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539360 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539392 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19c37710-4dfc-4e47-82dc-3622c2ace92c-serviceca\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539434 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e5e40a3-78e5-4616-9d69-45ae928aa773-iptables-alerter-script\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539463 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-multus-socket-dir-parent\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.540531 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539478 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-modprobe-d\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539529 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-systemd\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539558 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.539575 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539596 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539602 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539692 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e080c8a4-51cf-4821-bade-e8378e6690ce-konnectivity-ca\") pod \"konnectivity-agent-gznv2\" (UID: \"e080c8a4-51cf-4821-bade-e8378e6690ce\") " pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539758 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-kubelet\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539807 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-kubelet\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539831 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovnkube-config\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.539859 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs podName:97b8e7c6-8c74-4b67-bcb3-fa61744d2214 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:50.039766683 +0000 UTC m=+2.057653325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs") pod "network-metrics-daemon-zgbls" (UID: "97b8e7c6-8c74-4b67-bcb3-fa61744d2214") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539935 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-slash\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539964 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-cni-netd\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.539992 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-env-overrides\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540044 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-slash\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540064 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-cni-netd\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540080 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9dde853-d0e2-40da-bb68-ab7f67f456a7-tmp-dir\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.541355 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540106 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540137 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540182 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57d8172a-a0cd-4627-9e29-a03caae5da3c-tmp\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540211 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn52b\" (UniqueName: \"kubernetes.io/projected/c9dde853-d0e2-40da-bb68-ab7f67f456a7-kube-api-access-wn52b\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540235 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-cnibin\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540261 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-k8s-cni-cncf-io\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540285 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-device-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540331 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e080c8a4-51cf-4821-bade-e8378e6690ce-agent-certs\") pod \"konnectivity-agent-gznv2\" (UID: \"e080c8a4-51cf-4821-bade-e8378e6690ce\") " pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540354 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovn-node-metrics-cert\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540397 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysconfig\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540426 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdc8\" (UniqueName: \"kubernetes.io/projected/57d8172a-a0cd-4627-9e29-a03caae5da3c-kube-api-access-zhdc8\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540431 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b616b9b-a4ac-4feb-a082-6eae00565d60-env-overrides\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540454 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-cni-bin\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540432 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9dde853-d0e2-40da-bb68-ab7f67f456a7-tmp-dir\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540475 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-device-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-cni-bin\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540515 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-etc-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.542351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540519 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-cni-bin\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540534 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysconfig\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540541 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-cni-multus\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/540f8401-c269-48d4-8475-00950f61fd03-multus-daemon-config\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540585 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-cni-bin\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540597 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brlk9\" (UniqueName: \"kubernetes.io/projected/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-kube-api-access-brlk9\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540634 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-k8s-cni-cncf-io\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540651 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-system-cni-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-systemd\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540696 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-system-cni-dir\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540699 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540744 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-var-lib-cni-multus\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540770 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-cnibin\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.540971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2wk\" (UniqueName: \"kubernetes.io/projected/2b616b9b-a4ac-4feb-a082-6eae00565d60-kube-api-access-ph2wk\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541008 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/540f8401-c269-48d4-8475-00950f61fd03-cni-binary-copy\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541036 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541066 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-multus-certs\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541073 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-etc-openvswitch\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.543209 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541093 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/110693d2-151e-459c-8789-094e101446fc-dbus\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541284 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541334 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-host-run-multus-certs\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/540f8401-c269-48d4-8475-00950f61fd03-multus-daemon-config\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541549 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/540f8401-c269-48d4-8475-00950f61fd03-cni-binary-copy\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-socket-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541628 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/110693d2-151e-459c-8789-094e101446fc-dbus\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541669 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19c37710-4dfc-4e47-82dc-3622c2ace92c-host\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541699 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-system-cni-dir\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541728 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-cni-binary-copy\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541753 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-os-release\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541782 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-socket-dir\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541788 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-etc-kubernetes\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541833 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/110693d2-151e-459c-8789-094e101446fc-kubelet-config\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541842 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-etc-kubernetes\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541853 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19c37710-4dfc-4e47-82dc-3622c2ace92c-host\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541883 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9dde853-d0e2-40da-bb68-ab7f67f456a7-hosts-file\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.543940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541889 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/110693d2-151e-459c-8789-094e101446fc-kubelet-config\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541916 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2xd\" (UniqueName: \"kubernetes.io/projected/19c37710-4dfc-4e47-82dc-3622c2ace92c-kube-api-access-gr2xd\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541930 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-run-systemd\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541945 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5p7x\" (UniqueName: \"kubernetes.io/projected/15efe522-b59d-40f5-982b-00b752ea3fa6-kube-api-access-n5p7x\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541977 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-run-netns\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541989 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9dde853-d0e2-40da-bb68-ab7f67f456a7-hosts-file\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541950 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.541992 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/540f8401-c269-48d4-8475-00950f61fd03-os-release\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542081 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-system-cni-dir\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysctl-d\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542639 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwqb\" (UniqueName: \"kubernetes.io/projected/540f8401-c269-48d4-8475-00950f61fd03-kube-api-access-tpwqb\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542175 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-sysctl-d\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542691 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542718 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-node-log\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542550 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15efe522-b59d-40f5-982b-00b752ea3fa6-cni-binary-copy\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542759 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-cnibin\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.542838 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542849 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15efe522-b59d-40f5-982b-00b752ea3fa6-cnibin\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.544520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542202 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-host-run-netns\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.545068 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.542895 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b616b9b-a4ac-4feb-a082-6eae00565d60-node-log\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.545068 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.542903 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret podName:110693d2-151e-459c-8789-094e101446fc nodeName:}" failed. No retries permitted until 2026-07-23 11:26:50.042881303 +0000 UTC m=+2.060767925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret") pod "global-pull-secret-syncer-4bqk7" (UID: "110693d2-151e-459c-8789-094e101446fc") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:49.545068 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.543111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57d8172a-a0cd-4627-9e29-a03caae5da3c-etc-tuned\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.545068 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.543459 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57d8172a-a0cd-4627-9e29-a03caae5da3c-tmp\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.545068 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.543742 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b616b9b-a4ac-4feb-a082-6eae00565d60-ovn-node-metrics-cert\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.545068 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.543761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e080c8a4-51cf-4821-bade-e8378e6690ce-agent-certs\") pod \"konnectivity-agent-gznv2\" (UID: \"e080c8a4-51cf-4821-bade-e8378e6690ce\") " pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:26:49.547140 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.547116 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:49.547215 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.547146 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:49.547215 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.547160 2556 projected.go:196] Error preparing data for projected volume kube-api-access-9n8ss for pod openshift-network-diagnostics/network-check-target-tn5xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:49.547330 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:49.547248 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss podName:8b17bb8b-f1a8-468e-b1fb-64f534c30019 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:50.047225076 +0000 UTC m=+2.065111713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9n8ss" (UniqueName: "kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss") pod "network-check-target-tn5xd" (UID: "8b17bb8b-f1a8-468e-b1fb-64f534c30019") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:49.550607 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.550566 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58ct\" (UniqueName: \"kubernetes.io/projected/1e5e40a3-78e5-4616-9d69-45ae928aa773-kube-api-access-p58ct\") pod \"iptables-alerter-v9hgw\" (UID: \"1e5e40a3-78e5-4616-9d69-45ae928aa773\") " pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.551031 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.551000 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdc8\" (UniqueName: \"kubernetes.io/projected/57d8172a-a0cd-4627-9e29-a03caae5da3c-kube-api-access-zhdc8\") pod \"tuned-zgxd4\" (UID: \"57d8172a-a0cd-4627-9e29-a03caae5da3c\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.551345 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.551311 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9n2\" (UniqueName: \"kubernetes.io/projected/ead18787-8d08-4b98-b51d-7a1ed91ed1e1-kube-api-access-pm9n2\") pod \"aws-ebs-csi-driver-node-t67bs\" (UID: \"ead18787-8d08-4b98-b51d-7a1ed91ed1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.552027 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.551795 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2xd\" (UniqueName: \"kubernetes.io/projected/19c37710-4dfc-4e47-82dc-3622c2ace92c-kube-api-access-gr2xd\") pod \"node-ca-r5v5q\" (UID: \"19c37710-4dfc-4e47-82dc-3622c2ace92c\") " pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.552027 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.551797 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn52b\" (UniqueName: \"kubernetes.io/projected/c9dde853-d0e2-40da-bb68-ab7f67f456a7-kube-api-access-wn52b\") pod \"node-resolver-bxgc5\" (UID: \"c9dde853-d0e2-40da-bb68-ab7f67f456a7\") " pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.552027 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.551968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2wk\" (UniqueName: \"kubernetes.io/projected/2b616b9b-a4ac-4feb-a082-6eae00565d60-kube-api-access-ph2wk\") pod \"ovnkube-node-mnjnv\" (UID: \"2b616b9b-a4ac-4feb-a082-6eae00565d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.552720 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.552700 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwqb\" (UniqueName: \"kubernetes.io/projected/540f8401-c269-48d4-8475-00950f61fd03-kube-api-access-tpwqb\") pod \"multus-b52vr\" (UID: \"540f8401-c269-48d4-8475-00950f61fd03\") " pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.552934 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.552912 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brlk9\" (UniqueName: \"kubernetes.io/projected/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-kube-api-access-brlk9\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:49.553593 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.553564 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5p7x\" (UniqueName: \"kubernetes.io/projected/15efe522-b59d-40f5-982b-00b752ea3fa6-kube-api-access-n5p7x\") pod \"multus-additional-cni-plugins-n952k\" (UID: \"15efe522-b59d-40f5-982b-00b752ea3fa6\") " pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.665335 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.665227 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5v5q" Jul 23 11:26:49.672697 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.672669 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n952k" Jul 23 11:26:49.691585 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.691543 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b52vr" Jul 23 11:26:49.707668 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.707632 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:26:49.715788 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.715759 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:49.731108 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.731067 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" Jul 23 11:26:49.752896 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.752855 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" Jul 23 11:26:49.769139 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.769103 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxgc5" Jul 23 11:26:49.784804 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.784767 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9hgw" Jul 23 11:26:49.790106 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.790077 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:26:49.840504 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:49.840459 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fc63d6b59e151c5bd785e99c9b32e9.slice/crio-13efc2b4a6b4d726225604b655448708f410ba683f68b018418a7416d3516121 WatchSource:0}: Error finding container 13efc2b4a6b4d726225604b655448708f410ba683f68b018418a7416d3516121: Status 404 returned error can't find the container with id 13efc2b4a6b4d726225604b655448708f410ba683f68b018418a7416d3516121 Jul 23 11:26:49.840759 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:49.840736 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202d9bf1b3a2ad64be78b9941f522638.slice/crio-22ee56ef73e969bd174955e9d449ec59d180bbd919614130964e962812f1375f WatchSource:0}: Error finding container 22ee56ef73e969bd174955e9d449ec59d180bbd919614130964e962812f1375f: Status 404 returned error can't find the container with id 22ee56ef73e969bd174955e9d449ec59d180bbd919614130964e962812f1375f Jul 23 11:26:49.846666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:49.846629 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:26:50.047744 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.047642 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:50.047744 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.047687 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:50.047744 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.047724 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:50.048038 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.047828 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:50.048038 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.047849 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:50.048038 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.047872 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:50.048038 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.047887 2556 projected.go:196] Error preparing data for projected volume kube-api-access-9n8ss for pod openshift-network-diagnostics/network-check-target-tn5xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:50.048038 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.047900 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs podName:97b8e7c6-8c74-4b67-bcb3-fa61744d2214 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:51.047881959 +0000 UTC m=+3.065768574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs") pod "network-metrics-daemon-zgbls" (UID: "97b8e7c6-8c74-4b67-bcb3-fa61744d2214") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:50.048038 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.047828 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:50.048038 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.047939 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss podName:8b17bb8b-f1a8-468e-b1fb-64f534c30019 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:51.047923471 +0000 UTC m=+3.065810088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n8ss" (UniqueName: "kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss") pod "network-check-target-tn5xd" (UID: "8b17bb8b-f1a8-468e-b1fb-64f534c30019") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:50.048038 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.047970 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret podName:110693d2-151e-459c-8789-094e101446fc nodeName:}" failed. No retries permitted until 2026-07-23 11:26:51.047957213 +0000 UTC m=+3.065843830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret") pod "global-pull-secret-syncer-4bqk7" (UID: "110693d2-151e-459c-8789-094e101446fc") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:50.145087 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:50.145052 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c37710_4dfc_4e47_82dc_3622c2ace92c.slice/crio-68802c3a16f69318709bd2556dcdf02f22a9a9d2307ecdbe6ad26fcffbf1de97 WatchSource:0}: Error finding container 68802c3a16f69318709bd2556dcdf02f22a9a9d2307ecdbe6ad26fcffbf1de97: Status 404 returned error can't find the container with id 68802c3a16f69318709bd2556dcdf02f22a9a9d2307ecdbe6ad26fcffbf1de97 Jul 23 11:26:50.226122 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:50.226079 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b616b9b_a4ac_4feb_a082_6eae00565d60.slice/crio-5fcdc31ef1e5d466f929b31d028ab21e717849a4a7cbcb6a133a5b3636e4df0e WatchSource:0}: Error finding container 5fcdc31ef1e5d466f929b31d028ab21e717849a4a7cbcb6a133a5b3636e4df0e: Status 404 returned error can't find the container with id 5fcdc31ef1e5d466f929b31d028ab21e717849a4a7cbcb6a133a5b3636e4df0e Jul 23 11:26:50.240117 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:50.240084 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode080c8a4_51cf_4821_bade_e8378e6690ce.slice/crio-fc8a44c1e979fab0e4b986d67fe71ffe7b50c76fcfd0e5430fb677b28d2b2c8f WatchSource:0}: Error finding container fc8a44c1e979fab0e4b986d67fe71ffe7b50c76fcfd0e5430fb677b28d2b2c8f: Status 404 returned error can't find the container with id fc8a44c1e979fab0e4b986d67fe71ffe7b50c76fcfd0e5430fb677b28d2b2c8f Jul 23 11:26:50.246511 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:50.246479 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d8172a_a0cd_4627_9e29_a03caae5da3c.slice/crio-43fd4f16138cb01a8cdcaa5f7c8d2e5fa9c424cb388ccf9f42b22faed36ac0d3 WatchSource:0}: Error finding container 43fd4f16138cb01a8cdcaa5f7c8d2e5fa9c424cb388ccf9f42b22faed36ac0d3: Status 404 returned error can't find the container with id 43fd4f16138cb01a8cdcaa5f7c8d2e5fa9c424cb388ccf9f42b22faed36ac0d3 Jul 23 11:26:50.250442 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:50.250408 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead18787_8d08_4b98_b51d_7a1ed91ed1e1.slice/crio-8df99212849f9aaac2b2b8a75de076f1700e0a3e5a8d15535503fe6822e8c7a6 WatchSource:0}: Error finding container 8df99212849f9aaac2b2b8a75de076f1700e0a3e5a8d15535503fe6822e8c7a6: Status 404 returned error can't find the container with id 8df99212849f9aaac2b2b8a75de076f1700e0a3e5a8d15535503fe6822e8c7a6 Jul 23 11:26:50.269596 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:50.269548 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5e40a3_78e5_4616_9d69_45ae928aa773.slice/crio-15ca7f5bb666783f69870de1ffe4b2948531cf99297e045914a2b7f6ed9bb595 WatchSource:0}: Error finding container 15ca7f5bb666783f69870de1ffe4b2948531cf99297e045914a2b7f6ed9bb595: Status 404 returned error can't find the container with id 15ca7f5bb666783f69870de1ffe4b2948531cf99297e045914a2b7f6ed9bb595 Jul 23 11:26:50.378849 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.378785 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:21:50 +0000 UTC" deadline="2028-01-05 15:42:14.972571469 +0000 UTC" Jul 23 11:26:50.378849 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.378841 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12748h15m24.593733363s" Jul 23 11:26:50.463992 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.463961 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:50.489133 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:26:50.489092 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15efe522_b59d_40f5_982b_00b752ea3fa6.slice/crio-ef1cd7324ba4de1666d071d68049f0a1083bbe270b0852892d503e5eb0745f82 WatchSource:0}: Error finding container ef1cd7324ba4de1666d071d68049f0a1083bbe270b0852892d503e5eb0745f82: Status 404 returned error can't find the container with id ef1cd7324ba4de1666d071d68049f0a1083bbe270b0852892d503e5eb0745f82 Jul 23 11:26:50.494169 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.494137 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:50.494314 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.494286 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:26:50.494378 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.494340 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:50.494435 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:50.494421 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:26:50.499379 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.499304 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" event={"ID":"ead18787-8d08-4b98-b51d-7a1ed91ed1e1","Type":"ContainerStarted","Data":"8df99212849f9aaac2b2b8a75de076f1700e0a3e5a8d15535503fe6822e8c7a6"} Jul 23 11:26:50.500479 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.500447 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxgc5" event={"ID":"c9dde853-d0e2-40da-bb68-ab7f67f456a7","Type":"ContainerStarted","Data":"84c298132868f37c2782dafb829ae2fbe935c618c1f5f6fbe376ffa812397548"} Jul 23 11:26:50.501765 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.501739 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5v5q" event={"ID":"19c37710-4dfc-4e47-82dc-3622c2ace92c","Type":"ContainerStarted","Data":"68802c3a16f69318709bd2556dcdf02f22a9a9d2307ecdbe6ad26fcffbf1de97"} Jul 23 11:26:50.502965 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.502937 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b52vr" event={"ID":"540f8401-c269-48d4-8475-00950f61fd03","Type":"ContainerStarted","Data":"9656a5bbad317321f8910d7c1acd0dc8350f4b67817bf1501951559d7bbe24d3"} Jul 23 11:26:50.504287 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.504254 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" event={"ID":"34fc63d6b59e151c5bd785e99c9b32e9","Type":"ContainerStarted","Data":"13efc2b4a6b4d726225604b655448708f410ba683f68b018418a7416d3516121"} Jul 23 11:26:50.505714 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.505685 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" event={"ID":"57d8172a-a0cd-4627-9e29-a03caae5da3c","Type":"ContainerStarted","Data":"43fd4f16138cb01a8cdcaa5f7c8d2e5fa9c424cb388ccf9f42b22faed36ac0d3"} Jul 23 11:26:50.507020 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.506991 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gznv2" event={"ID":"e080c8a4-51cf-4821-bade-e8378e6690ce","Type":"ContainerStarted","Data":"fc8a44c1e979fab0e4b986d67fe71ffe7b50c76fcfd0e5430fb677b28d2b2c8f"} Jul 23 11:26:50.508473 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.508446 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"5fcdc31ef1e5d466f929b31d028ab21e717849a4a7cbcb6a133a5b3636e4df0e"} Jul 23 11:26:50.509733 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.509706 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" event={"ID":"202d9bf1b3a2ad64be78b9941f522638","Type":"ContainerStarted","Data":"22ee56ef73e969bd174955e9d449ec59d180bbd919614130964e962812f1375f"} Jul 23 11:26:50.511110 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.511084 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n952k" event={"ID":"15efe522-b59d-40f5-982b-00b752ea3fa6","Type":"ContainerStarted","Data":"ef1cd7324ba4de1666d071d68049f0a1083bbe270b0852892d503e5eb0745f82"} Jul 23 11:26:50.512375 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:50.512352 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9hgw" event={"ID":"1e5e40a3-78e5-4616-9d69-45ae928aa773","Type":"ContainerStarted","Data":"15ca7f5bb666783f69870de1ffe4b2948531cf99297e045914a2b7f6ed9bb595"} Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:51.056684 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:51.056770 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:51.056805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.056968 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.057035 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs podName:97b8e7c6-8c74-4b67-bcb3-fa61744d2214 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:53.057015563 +0000 UTC m=+5.074902184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs") pod "network-metrics-daemon-zgbls" (UID: "97b8e7c6-8c74-4b67-bcb3-fa61744d2214") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.057460 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.057513 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret podName:110693d2-151e-459c-8789-094e101446fc nodeName:}" failed. No retries permitted until 2026-07-23 11:26:53.05749714 +0000 UTC m=+5.075383764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret") pod "global-pull-secret-syncer-4bqk7" (UID: "110693d2-151e-459c-8789-094e101446fc") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.057602 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.057616 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.057630 2556 projected.go:196] Error preparing data for projected volume kube-api-access-9n8ss for pod openshift-network-diagnostics/network-check-target-tn5xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:51.057741 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.057685 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss podName:8b17bb8b-f1a8-468e-b1fb-64f534c30019 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:53.0576735 +0000 UTC m=+5.075560120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n8ss" (UniqueName: "kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss") pod "network-check-target-tn5xd" (UID: "8b17bb8b-f1a8-468e-b1fb-64f534c30019") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:51.380132 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:51.380018 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:21:50 +0000 UTC" deadline="2028-02-05 19:31:41.656836244 +0000 UTC" Jul 23 11:26:51.380132 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:51.380066 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13496h4m50.276775862s" Jul 23 11:26:51.494854 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:51.494814 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:51.495064 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:51.494982 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:26:51.657712 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:51.657615 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:52.496015 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:52.495711 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:52.496520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:52.495719 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:52.496520 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:52.496140 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:26:52.496520 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:52.496226 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:53.075685 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:53.075752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:53.075816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.075983 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.076058 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret podName:110693d2-151e-459c-8789-094e101446fc nodeName:}" failed. No retries permitted until 2026-07-23 11:26:57.076034964 +0000 UTC m=+9.093921605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret") pod "global-pull-secret-syncer-4bqk7" (UID: "110693d2-151e-459c-8789-094e101446fc") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.076548 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.076572 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.076587 2556 projected.go:196] Error preparing data for projected volume kube-api-access-9n8ss for pod openshift-network-diagnostics/network-check-target-tn5xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.076639 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss podName:8b17bb8b-f1a8-468e-b1fb-64f534c30019 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:57.07662218 +0000 UTC m=+9.094508798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n8ss" (UniqueName: "kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss") pod "network-check-target-tn5xd" (UID: "8b17bb8b-f1a8-468e-b1fb-64f534c30019") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.076759 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:53.076842 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.076802 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs podName:97b8e7c6-8c74-4b67-bcb3-fa61744d2214 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:57.076788556 +0000 UTC m=+9.094675179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs") pod "network-metrics-daemon-zgbls" (UID: "97b8e7c6-8c74-4b67-bcb3-fa61744d2214") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:53.495249 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:53.494539 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:53.495249 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:53.494727 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:26:54.496535 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:54.496487 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:54.497098 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:54.496666 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:26:54.497098 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:54.496699 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:54.497098 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:54.496833 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:26:55.495312 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:55.494709 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:55.495312 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:55.494883 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:26:56.497002 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:56.496960 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:56.497491 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:56.497117 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:26:56.497491 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:56.496960 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:56.497491 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:56.497196 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:57.113988 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:57.114066 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:57.114099 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.114176 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.114239 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.114257 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.114277 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.114291 2556 projected.go:196] Error preparing data for projected volume kube-api-access-9n8ss for pod openshift-network-diagnostics/network-check-target-tn5xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:57.114309 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.114260 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret podName:110693d2-151e-459c-8789-094e101446fc nodeName:}" failed. No retries permitted until 2026-07-23 11:27:05.114243148 +0000 UTC m=+17.132129761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret") pod "global-pull-secret-syncer-4bqk7" (UID: "110693d2-151e-459c-8789-094e101446fc") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:57.115022 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.114358 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs podName:97b8e7c6-8c74-4b67-bcb3-fa61744d2214 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:05.114338518 +0000 UTC m=+17.132225140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs") pod "network-metrics-daemon-zgbls" (UID: "97b8e7c6-8c74-4b67-bcb3-fa61744d2214") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:57.115022 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.114374 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss podName:8b17bb8b-f1a8-468e-b1fb-64f534c30019 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:05.114364975 +0000 UTC m=+17.132251599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n8ss" (UniqueName: "kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss") pod "network-check-target-tn5xd" (UID: "8b17bb8b-f1a8-468e-b1fb-64f534c30019") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:57.494780 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:57.494681 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:57.494962 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:57.494846 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:26:58.495364 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:58.495325 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:26:58.495892 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:58.495420 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:26:58.495892 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:58.495458 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:26:58.495892 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:58.495547 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:26:59.494818 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:26:59.494778 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:26:59.495028 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:26:59.494931 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:00.495246 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:00.494770 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:00.495246 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:00.494787 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:00.495246 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:00.494917 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:00.495246 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:00.495058 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:01.494206 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:01.494171 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:01.494421 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:01.494305 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:02.495143 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:02.494620 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:02.495143 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:02.494750 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:02.495143 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:02.494804 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:02.495143 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:02.494900 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:03.494237 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:03.494198 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:03.494433 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:03.494346 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:04.494483 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:04.494382 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:04.494976 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:04.494532 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:04.494976 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:04.494600 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:04.494976 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:04.494719 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:05.177744 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:05.177705 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:05.177744 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:05.177767 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:05.178026 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.177888 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:27:05.178026 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.177888 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:27:05.178026 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.177904 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:27:05.178026 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.177920 2556 projected.go:196] Error preparing data for projected volume kube-api-access-9n8ss for pod openshift-network-diagnostics/network-check-target-tn5xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:27:05.178026 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.177959 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:27:05.178026 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:05.177899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:05.178026 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.177962 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret podName:110693d2-151e-459c-8789-094e101446fc nodeName:}" failed. No retries permitted until 2026-07-23 11:27:21.177942489 +0000 UTC m=+33.195829102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret") pod "global-pull-secret-syncer-4bqk7" (UID: "110693d2-151e-459c-8789-094e101446fc") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:27:05.178026 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.178017 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss podName:8b17bb8b-f1a8-468e-b1fb-64f534c30019 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:21.178000089 +0000 UTC m=+33.195886701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n8ss" (UniqueName: "kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss") pod "network-check-target-tn5xd" (UID: "8b17bb8b-f1a8-468e-b1fb-64f534c30019") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:27:05.178431 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.178039 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs podName:97b8e7c6-8c74-4b67-bcb3-fa61744d2214 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:21.178029777 +0000 UTC m=+33.195916392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs") pod "network-metrics-daemon-zgbls" (UID: "97b8e7c6-8c74-4b67-bcb3-fa61744d2214") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:27:05.494140 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:05.494056 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:05.494315 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:05.494211 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:06.494868 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:06.494826 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:06.495348 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:06.494958 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:06.495348 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:06.495019 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:06.495348 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:06.495144 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:07.494441 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:07.494398 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:07.494677 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:07.494548 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:08.495272 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:08.495225 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:08.495701 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:08.495382 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:08.495701 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:08.495418 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:08.495701 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:08.495523 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:09.494746 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:09.494704 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:09.494971 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:09.494861 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:10.494379 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:10.494342 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:10.494980 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:10.494343 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:10.494980 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:10.494451 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:10.494980 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:10.494625 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:11.494589 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:11.494544 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:11.495072 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:11.494698 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:11.574108 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:11.574072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b52vr" event={"ID":"540f8401-c269-48d4-8475-00950f61fd03","Type":"ContainerStarted","Data":"cbadf2d17f09923e124e2d317d03579e05a6c795beb6e584903689de9eddfc37"} Jul 23 11:27:11.576698 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:11.576668 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" event={"ID":"57d8172a-a0cd-4627-9e29-a03caae5da3c","Type":"ContainerStarted","Data":"f1e5e77fd522c7af4bf2a58f3e7cfc8edd056d9970a1fb15dc9911d7529464b9"} Jul 23 11:27:11.579504 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:11.579351 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"5de1945e1e2aa89c95a3fc00f5690d6721aedd7bb4ed0477a6e984602aa97852"} Jul 23 11:27:11.579504 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:11.579446 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"38c1a565922e608f5d3be5d25b8c5126f986058fde2d28ed28e1d360b1c90834"} Jul 23 11:27:11.583513 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:11.583482 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" event={"ID":"202d9bf1b3a2ad64be78b9941f522638","Type":"ContainerStarted","Data":"b15507ea1c554ac59adbab4efff92c7d1671123c68a49f8a092f253ab9dbf5aa"} Jul 23 11:27:11.596322 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:11.596266 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b52vr" podStartSLOduration=1.623982016 podStartE2EDuration="22.596248515s" totalImagesPullingTime="20.972266499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:11.595607019 +0000 UTC m=+23.613493652" watchObservedRunningTime="2026-07-23 11:27:11.596248515 +0000 UTC m=+23.614135128" Jul 23 11:27:11.609656 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:11.609605 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" podStartSLOduration=22.609562768 podStartE2EDuration="22.609562768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:11.609154675 +0000 UTC m=+23.627041309" watchObservedRunningTime="2026-07-23 11:27:11.609562768 +0000 UTC m=+23.627449403" Jul 23 11:27:12.495099 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.494629 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:12.495773 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.494660 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:12.495773 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:12.495217 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:12.495773 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:12.495251 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:12.587069 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.587036 2556 generic.go:356] "Generic (PLEG): container finished" podID="15efe522-b59d-40f5-982b-00b752ea3fa6" containerID="3a3fb891fe3e3d20dcfbe55c0cd69ab438e8859cae6411301b8a63b4b96cba7d" exitCode=0 Jul 23 11:27:12.587273 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.587133 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n952k" event={"ID":"15efe522-b59d-40f5-982b-00b752ea3fa6","Type":"ContainerDied","Data":"3a3fb891fe3e3d20dcfbe55c0cd69ab438e8859cae6411301b8a63b4b96cba7d"} Jul 23 11:27:12.588822 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.588794 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" event={"ID":"ead18787-8d08-4b98-b51d-7a1ed91ed1e1","Type":"ContainerStarted","Data":"b46bf672c603b77aaef97a84bcec6ace4674bc49f50c5f47198fdde386fe8744"} Jul 23 11:27:12.590291 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.590245 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxgc5" event={"ID":"c9dde853-d0e2-40da-bb68-ab7f67f456a7","Type":"ContainerStarted","Data":"a49a4a09f544885da0b39cb3799b6a11158fd06516ff3c9fbc443cf14c6b9f97"} Jul 23 11:27:12.591848 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.591823 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5v5q" event={"ID":"19c37710-4dfc-4e47-82dc-3622c2ace92c","Type":"ContainerStarted","Data":"9ea0b0ceefdf2dac0858e43c4ffdbe71935b214aed19c242445a640c4f1f743c"} Jul 23 11:27:12.593376 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.593348 2556 generic.go:356] "Generic (PLEG): container finished" podID="34fc63d6b59e151c5bd785e99c9b32e9" containerID="872145c74f12d5631908e13b9ec3b2d458574c650ffaa600679681809c2cc036" exitCode=0 Jul 23 11:27:12.593483 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.593425 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" event={"ID":"34fc63d6b59e151c5bd785e99c9b32e9","Type":"ContainerDied","Data":"872145c74f12d5631908e13b9ec3b2d458574c650ffaa600679681809c2cc036"} Jul 23 11:27:12.595352 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.595325 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gznv2" event={"ID":"e080c8a4-51cf-4821-bade-e8378e6690ce","Type":"ContainerStarted","Data":"77c3045f8d2ecc43c2f388b59023c876b0a844ff678cc92f4f496e647afc1f8e"} Jul 23 11:27:12.598194 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.598176 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:27:12.598469 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.598452 2556 generic.go:356] "Generic (PLEG): container finished" podID="2b616b9b-a4ac-4feb-a082-6eae00565d60" containerID="5de1945e1e2aa89c95a3fc00f5690d6721aedd7bb4ed0477a6e984602aa97852" exitCode=1 Jul 23 11:27:12.598562 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.598539 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerDied","Data":"5de1945e1e2aa89c95a3fc00f5690d6721aedd7bb4ed0477a6e984602aa97852"} Jul 23 11:27:12.598645 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.598621 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"1cbda59992df3b6f67ce7a2a36351af571a7ff41429f5162e3b299b56c46a3ca"} Jul 23 11:27:12.598687 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.598670 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"02c30fc1b37ecb5bdbf830e3fb668c32337384d24ef75673d3071ed7abc9e170"} Jul 23 11:27:12.598732 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.598718 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"9d74389f9ecdd8702299ce342d41a380d69f968ce8bea09005da59b190dce3b5"} Jul 23 11:27:12.598788 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.598776 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"dfddd9cc7d007b1841f59c9fee0677e4c9e4b33770ecc400e60de6e92709e7f2"} Jul 23 11:27:12.613084 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.613038 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zgxd4" podStartSLOduration=2.832026031 podStartE2EDuration="23.613021922s" totalImagesPullingTime="20.780995891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:11.628396154 +0000 UTC m=+23.646282789" watchObservedRunningTime="2026-07-23 11:27:12.613021922 +0000 UTC m=+24.630908556" Jul 23 11:27:12.651919 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.651848 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bxgc5" podStartSLOduration=2.746040848 podStartE2EDuration="23.651826355s" totalImagesPullingTime="20.905785507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:12.651450912 +0000 UTC m=+24.669337549" watchObservedRunningTime="2026-07-23 11:27:12.651826355 +0000 UTC m=+24.669712991" Jul 23 11:27:12.687164 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.687115 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gznv2" podStartSLOduration=2.892717315 podStartE2EDuration="23.687095067s" totalImagesPullingTime="20.794377752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:12.670491212 +0000 UTC m=+24.688377846" watchObservedRunningTime="2026-07-23 11:27:12.687095067 +0000 UTC m=+24.704981733" Jul 23 11:27:12.687243 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:12.687204 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r5v5q" podStartSLOduration=2.7976921949999998 podStartE2EDuration="23.687198742s" totalImagesPullingTime="20.889506547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:12.686370449 +0000 UTC m=+24.704257087" watchObservedRunningTime="2026-07-23 11:27:12.687198742 +0000 UTC m=+24.705085377" Jul 23 11:27:13.374011 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.373980 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 11:27:13.407065 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.406957 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T11:27:13.37400427Z","UUID":"bb5cc26a-9216-492f-a741-80aa9e79db57","Handler":null,"Name":"","Endpoint":""} Jul 23 11:27:13.408757 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.408731 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 11:27:13.408904 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.408771 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 11:27:13.494028 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.493927 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:13.494212 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:13.494085 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:13.602482 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.602442 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9hgw" event={"ID":"1e5e40a3-78e5-4616-9d69-45ae928aa773","Type":"ContainerStarted","Data":"c3e4d0ec0a962f965c5ff88f9fc0b673560c11be204a78e024cfbb5a9dcea4ee"} Jul 23 11:27:13.604778 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.604743 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" event={"ID":"ead18787-8d08-4b98-b51d-7a1ed91ed1e1","Type":"ContainerStarted","Data":"c164ecc5e16b1f70679b4237716d3f13b878fffe9e96faf847b6af654309542c"} Jul 23 11:27:13.607213 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.607182 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" event={"ID":"34fc63d6b59e151c5bd785e99c9b32e9","Type":"ContainerStarted","Data":"b41a35b3e9b25c6c7c962f4da57c47af3db7da8a02cb25cae67a913007488f7d"} Jul 23 11:27:13.615926 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.615863 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v9hgw" podStartSLOduration=3.850726587 podStartE2EDuration="24.615838919s" totalImagesPullingTime="20.765112332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:13.614863346 +0000 UTC m=+25.632749981" watchObservedRunningTime="2026-07-23 11:27:13.615838919 +0000 UTC m=+25.633725555" Jul 23 11:27:13.629400 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.629343 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" podStartSLOduration=24.629326111 podStartE2EDuration="24.629326111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:13.628640413 +0000 UTC m=+25.646527048" watchObservedRunningTime="2026-07-23 11:27:13.629326111 +0000 UTC m=+25.647212744" Jul 23 11:27:13.939391 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.939351 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:27:13.939984 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:13.939960 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:27:14.451988 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.451917 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxgc5_c9dde853-d0e2-40da-bb68-ab7f67f456a7/dns-node-resolver/0.log" Jul 23 11:27:14.494281 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.494245 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:14.494478 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.494245 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:14.494478 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:14.494382 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:14.494628 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:14.494483 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:14.612055 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.612020 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" event={"ID":"ead18787-8d08-4b98-b51d-7a1ed91ed1e1","Type":"ContainerStarted","Data":"6facf5974970bea5c2cba406dff3e57680488050fd55e05a2cb59a3d9d97e192"} Jul 23 11:27:14.620751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.620725 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:27:14.621409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.621138 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"c2a74e4d60baccefc15fa52d488b7a3ca4737a5c47c712d4592142371af98776"} Jul 23 11:27:14.621409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.621264 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:27:14.621884 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.621865 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gznv2" Jul 23 11:27:14.630964 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:14.630902 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t67bs" podStartSLOduration=3.066329842 podStartE2EDuration="25.630884536s" totalImagesPullingTime="22.564554694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:14.630261851 +0000 UTC m=+26.648148488" watchObservedRunningTime="2026-07-23 11:27:14.630884536 +0000 UTC m=+26.648771170" Jul 23 11:27:15.494442 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:15.494406 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:15.494647 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:15.494521 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:15.631874 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:15.631843 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r5v5q_19c37710-4dfc-4e47-82dc-3622c2ace92c/node-ca/0.log" Jul 23 11:27:16.494846 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:16.494809 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:16.495019 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:16.494931 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:16.495019 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:16.494985 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:16.495101 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:16.495082 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:16.631399 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:16.631082 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:27:16.631661 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:16.631547 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"7dbcdb6c92f1f8dbc6551d11ee0f48098e9daa23ca341de74735b8d1af86deb0"} Jul 23 11:27:17.493873 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.493838 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:17.494674 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:17.493947 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:17.635246 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.635208 2556 generic.go:356] "Generic (PLEG): container finished" podID="15efe522-b59d-40f5-982b-00b752ea3fa6" containerID="7f2b7dfe52dc4d9144450214efd8deeff37130fddf7625abf3449335f8cbc20b" exitCode=0 Jul 23 11:27:17.635411 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.635261 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n952k" event={"ID":"15efe522-b59d-40f5-982b-00b752ea3fa6","Type":"ContainerDied","Data":"7f2b7dfe52dc4d9144450214efd8deeff37130fddf7625abf3449335f8cbc20b"} Jul 23 11:27:17.635537 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.635520 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:27:17.635722 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.635686 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:27:17.635842 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.635776 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:27:17.635842 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.635700 2556 scope.go:117] "RemoveContainer" containerID="5de1945e1e2aa89c95a3fc00f5690d6721aedd7bb4ed0477a6e984602aa97852" Jul 23 11:27:17.653673 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.653635 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:27:17.654016 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:17.653993 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:27:18.495113 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:18.494922 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:18.495692 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:18.495007 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:18.495692 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:18.495177 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:18.495692 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:18.495308 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:18.639926 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:18.639842 2556 generic.go:356] "Generic (PLEG): container finished" podID="15efe522-b59d-40f5-982b-00b752ea3fa6" containerID="46fac5402f689adb8c335b6d871fea0038218b520899e53d3f8d12721fd835a5" exitCode=0 Jul 23 11:27:18.639926 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:18.639888 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n952k" event={"ID":"15efe522-b59d-40f5-982b-00b752ea3fa6","Type":"ContainerDied","Data":"46fac5402f689adb8c335b6d871fea0038218b520899e53d3f8d12721fd835a5"} Jul 23 11:27:18.644168 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:18.644144 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:27:18.644529 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:18.644507 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" event={"ID":"2b616b9b-a4ac-4feb-a082-6eae00565d60","Type":"ContainerStarted","Data":"48c98d32d30923890ad0e22460d4b1f42aeef0f658f5ce69382435d461893eb3"} Jul 23 11:27:18.696053 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:18.695984 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" podStartSLOduration=8.790089182 podStartE2EDuration="29.695968708s" totalImagesPullingTime="20.905879526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:18.695319295 +0000 UTC m=+30.713205930" watchObservedRunningTime="2026-07-23 11:27:18.695968708 +0000 UTC m=+30.713855342" Jul 23 11:27:19.494288 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:19.494259 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:19.494459 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:19.494398 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:20.494775 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:20.494736 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:20.495216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:20.494750 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:20.495216 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:20.494851 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:20.495216 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:20.494943 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:20.651953 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:20.651910 2556 generic.go:356] "Generic (PLEG): container finished" podID="15efe522-b59d-40f5-982b-00b752ea3fa6" containerID="20847d68b9b1a307bcbe26961724c24b5bbd3436fdd4050a023ee22550ea1f70" exitCode=0 Jul 23 11:27:20.652142 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:20.651970 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n952k" event={"ID":"15efe522-b59d-40f5-982b-00b752ea3fa6","Type":"ContainerDied","Data":"20847d68b9b1a307bcbe26961724c24b5bbd3436fdd4050a023ee22550ea1f70"} Jul 23 11:27:21.194250 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:21.194202 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:21.194445 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:21.194277 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:21.194445 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:21.194307 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:21.194445 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.194365 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:27:21.194445 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.194431 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:27:21.194445 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.194443 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:27:21.194671 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.194466 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:27:21.194671 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.194479 2556 projected.go:196] Error preparing data for projected volume kube-api-access-9n8ss for pod openshift-network-diagnostics/network-check-target-tn5xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:27:21.194671 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.194445 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret podName:110693d2-151e-459c-8789-094e101446fc nodeName:}" failed. No retries permitted until 2026-07-23 11:27:53.194423834 +0000 UTC m=+65.212310461 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret") pod "global-pull-secret-syncer-4bqk7" (UID: "110693d2-151e-459c-8789-094e101446fc") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:27:21.194671 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.194529 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs podName:97b8e7c6-8c74-4b67-bcb3-fa61744d2214 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:53.194516183 +0000 UTC m=+65.212402904 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs") pod "network-metrics-daemon-zgbls" (UID: "97b8e7c6-8c74-4b67-bcb3-fa61744d2214") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:27:21.194671 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.194545 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss podName:8b17bb8b-f1a8-468e-b1fb-64f534c30019 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:53.194536139 +0000 UTC m=+65.212422758 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n8ss" (UniqueName: "kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss") pod "network-check-target-tn5xd" (UID: "8b17bb8b-f1a8-468e-b1fb-64f534c30019") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:27:21.494811 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:21.494705 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:21.495213 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:21.494891 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:22.494719 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:22.494683 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:22.494918 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:22.494817 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:22.494918 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:22.494861 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:22.495307 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:22.494937 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:23.494788 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:23.494747 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:23.494980 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:23.494899 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:24.494484 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:24.494446 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:24.494700 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:24.494446 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:24.494700 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:24.494560 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:24.494700 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:24.494638 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:25.494602 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:25.494495 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:25.494938 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:25.494628 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:25.668611 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:25.668559 2556 generic.go:356] "Generic (PLEG): container finished" podID="15efe522-b59d-40f5-982b-00b752ea3fa6" containerID="647ad5b74bd962d869217a744c8bc383054b4128a06654f911148d64c84460ae" exitCode=0 Jul 23 11:27:25.668611 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:25.668602 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n952k" event={"ID":"15efe522-b59d-40f5-982b-00b752ea3fa6","Type":"ContainerDied","Data":"647ad5b74bd962d869217a744c8bc383054b4128a06654f911148d64c84460ae"} Jul 23 11:27:26.497687 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:26.497654 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:26.498067 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:26.497756 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:26.498067 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:26.497807 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:26.498067 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:26.497890 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:26.674141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:26.674102 2556 generic.go:356] "Generic (PLEG): container finished" podID="15efe522-b59d-40f5-982b-00b752ea3fa6" containerID="aecc6dca0fa940e8e80aca9095b5826aff795306147cbb3fbcb6765551c2cb6a" exitCode=0 Jul 23 11:27:26.674278 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:26.674175 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n952k" event={"ID":"15efe522-b59d-40f5-982b-00b752ea3fa6","Type":"ContainerDied","Data":"aecc6dca0fa940e8e80aca9095b5826aff795306147cbb3fbcb6765551c2cb6a"} Jul 23 11:27:27.494243 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:27.494200 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:27.494425 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:27.494341 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:27.680167 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:27.680131 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n952k" event={"ID":"15efe522-b59d-40f5-982b-00b752ea3fa6","Type":"ContainerStarted","Data":"6ac03c08f387660d3322818fe905b34e873991f2c85bbaf0a5b81c29840cd29f"} Jul 23 11:27:27.702325 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:27.702274 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n952k" podStartSLOduration=7.723718173 podStartE2EDuration="38.702258736s" totalImagesPullingTime="30.978540563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:27.701395694 +0000 UTC m=+39.719282329" watchObservedRunningTime="2026-07-23 11:27:27.702258736 +0000 UTC m=+39.720145371" Jul 23 11:27:28.495485 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:28.495446 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:28.495713 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:28.495524 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:28.495713 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:28.495536 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:28.495713 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:28.495655 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:29.494089 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:29.494057 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:29.494486 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:29.494163 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:30.494723 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:30.494686 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:30.495118 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:30.494722 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:30.495118 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:30.494795 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:30.495118 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:30.494907 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:31.493898 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:31.493853 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:31.494104 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:31.493973 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:32.494480 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:32.494441 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:32.494902 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:32.494556 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:32.494902 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:32.494646 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:32.494902 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:32.494755 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:33.494884 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:33.494846 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:33.495375 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:33.494970 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:34.497897 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:34.497867 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:34.498356 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:34.497873 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:34.498356 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:34.497964 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:34.498356 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:34.498069 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:35.494703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:35.494663 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:35.494886 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:35.494775 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:36.493977 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:36.493944 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:36.494475 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:36.493944 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:36.494475 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:36.494045 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:36.494475 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:36.494135 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:37.494230 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:37.494044 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:37.494630 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:37.494317 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:37.834717 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:37.834680 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4bqk7"] Jul 23 11:27:37.834920 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:37.834805 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:37.834992 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:37.834935 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:37.836669 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:37.836628 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tn5xd"] Jul 23 11:27:37.836806 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:37.836748 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:37.836874 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:37.836855 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:37.837222 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:37.837197 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zgbls"] Jul 23 11:27:37.837305 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:37.837295 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:37.837402 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:37.837382 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:39.493957 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:39.493919 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:39.494732 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:39.493918 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:39.494732 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:39.494045 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn5xd" podUID="8b17bb8b-f1a8-468e-b1fb-64f534c30019" Jul 23 11:27:39.494732 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:39.493925 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:39.494732 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:39.494116 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zgbls" podUID="97b8e7c6-8c74-4b67-bcb3-fa61744d2214" Jul 23 11:27:39.494732 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:39.494184 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqk7" podUID="110693d2-151e-459c-8789-094e101446fc" Jul 23 11:27:41.346807 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.346778 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeReady" Jul 23 11:27:41.347190 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.346885 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 11:27:41.388045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.388008 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8q59n"] Jul 23 11:27:41.410438 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.410400 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fvjmm"] Jul 23 11:27:41.410649 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.410590 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.412852 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.412826 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 11:27:41.413661 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.413227 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d6qnx\"" Jul 23 11:27:41.413661 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.413546 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 11:27:41.426838 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.426763 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8q59n"] Jul 23 11:27:41.426838 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.426815 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fvjmm"] Jul 23 11:27:41.426838 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.426839 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4q8j9"] Jul 23 11:27:41.427053 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.426911 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fvjmm" Jul 23 11:27:41.430102 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.430070 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-89ttz\"" Jul 23 11:27:41.430272 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.430118 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 11:27:41.430272 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.430123 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 11:27:41.430272 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.430070 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 11:27:41.442796 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.442760 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4q8j9"] Jul 23 11:27:41.442962 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.442903 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.445182 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.445147 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 11:27:41.445358 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.445236 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfh54\"" Jul 23 11:27:41.445358 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.445273 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 11:27:41.445358 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.445292 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 11:27:41.445617 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.445520 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 11:27:41.494218 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.494173 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:41.494397 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.494183 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:41.494397 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.494194 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:41.496828 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.496801 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zh6tm\"" Jul 23 11:27:41.496955 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.496839 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 11:27:41.496955 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.496863 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 11:27:41.496955 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.496873 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 11:27:41.496955 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.496899 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8j6tk\"" Jul 23 11:27:41.497158 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.497143 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 11:27:41.547781 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.547743 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8580b584-f0d6-439a-b6d7-7e6d29ae82be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.547781 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.547781 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca94b96-042b-4db5-af4d-9855aec04c28-metrics-tls\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.547987 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.547802 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7mj\" (UniqueName: \"kubernetes.io/projected/1ca94b96-042b-4db5-af4d-9855aec04c28-kube-api-access-bc7mj\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.547987 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.547890 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm75x\" (UniqueName: \"kubernetes.io/projected/8580b584-f0d6-439a-b6d7-7e6d29ae82be-kube-api-access-vm75x\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.547987 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.547948 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca94b96-042b-4db5-af4d-9855aec04c28-tmp-dir\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.548095 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.547989 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8580b584-f0d6-439a-b6d7-7e6d29ae82be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.548095 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.548008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83c4d15b-687e-487a-8f65-12945db48eac-cert\") pod \"ingress-canary-fvjmm\" (UID: \"83c4d15b-687e-487a-8f65-12945db48eac\") " pod="openshift-ingress-canary/ingress-canary-fvjmm" Jul 23 11:27:41.548095 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.548026 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca94b96-042b-4db5-af4d-9855aec04c28-config-volume\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.548238 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.548140 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8580b584-f0d6-439a-b6d7-7e6d29ae82be-crio-socket\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.548238 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.548176 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7245x\" (UniqueName: \"kubernetes.io/projected/83c4d15b-687e-487a-8f65-12945db48eac-kube-api-access-7245x\") pod \"ingress-canary-fvjmm\" (UID: \"83c4d15b-687e-487a-8f65-12945db48eac\") " pod="openshift-ingress-canary/ingress-canary-fvjmm" Jul 23 11:27:41.548238 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.548204 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8580b584-f0d6-439a-b6d7-7e6d29ae82be-data-volume\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.648618 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.648551 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8580b584-f0d6-439a-b6d7-7e6d29ae82be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.648618 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.648622 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca94b96-042b-4db5-af4d-9855aec04c28-metrics-tls\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.648862 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.648647 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7mj\" (UniqueName: \"kubernetes.io/projected/1ca94b96-042b-4db5-af4d-9855aec04c28-kube-api-access-bc7mj\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.648862 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.648834 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm75x\" (UniqueName: \"kubernetes.io/projected/8580b584-f0d6-439a-b6d7-7e6d29ae82be-kube-api-access-vm75x\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.648961 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.648868 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca94b96-042b-4db5-af4d-9855aec04c28-tmp-dir\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.648961 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.648910 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8580b584-f0d6-439a-b6d7-7e6d29ae82be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.648961 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.648931 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83c4d15b-687e-487a-8f65-12945db48eac-cert\") pod \"ingress-canary-fvjmm\" (UID: \"83c4d15b-687e-487a-8f65-12945db48eac\") " pod="openshift-ingress-canary/ingress-canary-fvjmm" Jul 23 11:27:41.648961 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.648957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca94b96-042b-4db5-af4d-9855aec04c28-config-volume\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.649155 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.649019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8580b584-f0d6-439a-b6d7-7e6d29ae82be-crio-socket\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.649155 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.649043 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7245x\" (UniqueName: \"kubernetes.io/projected/83c4d15b-687e-487a-8f65-12945db48eac-kube-api-access-7245x\") pod \"ingress-canary-fvjmm\" (UID: \"83c4d15b-687e-487a-8f65-12945db48eac\") " pod="openshift-ingress-canary/ingress-canary-fvjmm" Jul 23 11:27:41.649155 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.649066 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8580b584-f0d6-439a-b6d7-7e6d29ae82be-data-volume\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.649293 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.649196 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca94b96-042b-4db5-af4d-9855aec04c28-tmp-dir\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.649293 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.649244 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8580b584-f0d6-439a-b6d7-7e6d29ae82be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.649397 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.649321 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8580b584-f0d6-439a-b6d7-7e6d29ae82be-crio-socket\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.649397 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.649387 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8580b584-f0d6-439a-b6d7-7e6d29ae82be-data-volume\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.649709 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.649691 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca94b96-042b-4db5-af4d-9855aec04c28-config-volume\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.653066 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.653036 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8580b584-f0d6-439a-b6d7-7e6d29ae82be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.653066 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.653069 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca94b96-042b-4db5-af4d-9855aec04c28-metrics-tls\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.653287 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.653194 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83c4d15b-687e-487a-8f65-12945db48eac-cert\") pod \"ingress-canary-fvjmm\" (UID: \"83c4d15b-687e-487a-8f65-12945db48eac\") " pod="openshift-ingress-canary/ingress-canary-fvjmm" Jul 23 11:27:41.656771 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.656735 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7mj\" (UniqueName: \"kubernetes.io/projected/1ca94b96-042b-4db5-af4d-9855aec04c28-kube-api-access-bc7mj\") pod \"dns-default-8q59n\" (UID: \"1ca94b96-042b-4db5-af4d-9855aec04c28\") " pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.656905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.656804 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm75x\" (UniqueName: \"kubernetes.io/projected/8580b584-f0d6-439a-b6d7-7e6d29ae82be-kube-api-access-vm75x\") pod \"insights-runtime-extractor-4q8j9\" (UID: \"8580b584-f0d6-439a-b6d7-7e6d29ae82be\") " pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.656905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.656872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7245x\" (UniqueName: \"kubernetes.io/projected/83c4d15b-687e-487a-8f65-12945db48eac-kube-api-access-7245x\") pod \"ingress-canary-fvjmm\" (UID: \"83c4d15b-687e-487a-8f65-12945db48eac\") " pod="openshift-ingress-canary/ingress-canary-fvjmm" Jul 23 11:27:41.722694 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.722602 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:41.736725 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.736694 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fvjmm" Jul 23 11:27:41.752517 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.752483 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4q8j9" Jul 23 11:27:41.944727 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.944500 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fvjmm"] Jul 23 11:27:41.947594 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.947542 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8q59n"] Jul 23 11:27:41.948716 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:41.948690 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c4d15b_687e_487a_8f65_12945db48eac.slice/crio-ea1e3e6e7ace3f378249f2ef64604fc759523d3658614a5324f5445d277a9fc9 WatchSource:0}: Error finding container ea1e3e6e7ace3f378249f2ef64604fc759523d3658614a5324f5445d277a9fc9: Status 404 returned error can't find the container with id ea1e3e6e7ace3f378249f2ef64604fc759523d3658614a5324f5445d277a9fc9 Jul 23 11:27:41.951384 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:41.951355 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca94b96_042b_4db5_af4d_9855aec04c28.slice/crio-ca7de7da19a0c47b5e7b9fcf14fb2be5d9d3f9ef0370ccc49a84270e0a797e22 WatchSource:0}: Error finding container ca7de7da19a0c47b5e7b9fcf14fb2be5d9d3f9ef0370ccc49a84270e0a797e22: Status 404 returned error can't find the container with id ca7de7da19a0c47b5e7b9fcf14fb2be5d9d3f9ef0370ccc49a84270e0a797e22 Jul 23 11:27:41.953125 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:41.953100 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4q8j9"] Jul 23 11:27:41.956910 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:41.956884 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8580b584_f0d6_439a_b6d7_7e6d29ae82be.slice/crio-7ad411d77fa4c79b8f9cfa2b1eb81814dd5499e0794aabcaae49bf237e00d6e0 WatchSource:0}: Error finding container 7ad411d77fa4c79b8f9cfa2b1eb81814dd5499e0794aabcaae49bf237e00d6e0: Status 404 returned error can't find the container with id 7ad411d77fa4c79b8f9cfa2b1eb81814dd5499e0794aabcaae49bf237e00d6e0 Jul 23 11:27:42.721171 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:42.721068 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q8j9" event={"ID":"8580b584-f0d6-439a-b6d7-7e6d29ae82be","Type":"ContainerStarted","Data":"7ad411d77fa4c79b8f9cfa2b1eb81814dd5499e0794aabcaae49bf237e00d6e0"} Jul 23 11:27:42.723675 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:42.723531 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fvjmm" event={"ID":"83c4d15b-687e-487a-8f65-12945db48eac","Type":"ContainerStarted","Data":"ea1e3e6e7ace3f378249f2ef64604fc759523d3658614a5324f5445d277a9fc9"} Jul 23 11:27:42.726674 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:42.726554 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8q59n" event={"ID":"1ca94b96-042b-4db5-af4d-9855aec04c28","Type":"ContainerStarted","Data":"ca7de7da19a0c47b5e7b9fcf14fb2be5d9d3f9ef0370ccc49a84270e0a797e22"} Jul 23 11:27:44.344446 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.344231 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c8fd7467-c8m8c"] Jul 23 11:27:44.346857 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.346828 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.349978 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.349949 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 11:27:44.350985 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.350960 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 11:27:44.351495 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.351132 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 11:27:44.351495 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.351324 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 11:27:44.351495 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.351374 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2dh7r\"" Jul 23 11:27:44.352036 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.352018 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 11:27:44.352198 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.352185 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 11:27:44.352392 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.352378 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 11:27:44.361699 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.361632 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c8fd7467-c8m8c"] Jul 23 11:27:44.451179 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.451137 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt"] Jul 23 11:27:44.453191 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.453162 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" Jul 23 11:27:44.455605 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.455562 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-r4v2c\"" Jul 23 11:27:44.455780 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.455615 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 23 11:27:44.462091 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.462004 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt"] Jul 23 11:27:44.473920 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.473879 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-service-ca\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.474106 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.473934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-serving-cert\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.474106 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.473970 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwd9n\" (UniqueName: \"kubernetes.io/projected/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-kube-api-access-qwd9n\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.474106 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.473998 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-oauth-serving-cert\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.474106 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.474097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-oauth-config\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.474282 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.474131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-config\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.575233 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.575189 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-oauth-config\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.575431 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.575242 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-config\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.575501 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.575479 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-service-ca\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.575558 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.575513 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0a4cd02f-2573-45e8-a321-5de35dc4c38f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-g4lwt\" (UID: \"0a4cd02f-2573-45e8-a321-5de35dc4c38f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" Jul 23 11:27:44.575558 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.575538 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-serving-cert\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.575735 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.575588 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwd9n\" (UniqueName: \"kubernetes.io/projected/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-kube-api-access-qwd9n\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.575735 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.575656 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-oauth-serving-cert\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.576621 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.576337 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-service-ca\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.576621 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.576374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-config\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.576621 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.576374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-oauth-serving-cert\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.586618 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.582662 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-oauth-config\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.586618 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.584989 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwd9n\" (UniqueName: \"kubernetes.io/projected/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-kube-api-access-qwd9n\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.593486 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.593449 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-serving-cert\") pod \"console-78c8fd7467-c8m8c\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.661747 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.661655 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:44.677099 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.677059 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0a4cd02f-2573-45e8-a321-5de35dc4c38f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-g4lwt\" (UID: \"0a4cd02f-2573-45e8-a321-5de35dc4c38f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" Jul 23 11:27:44.686820 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.686790 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0a4cd02f-2573-45e8-a321-5de35dc4c38f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-g4lwt\" (UID: \"0a4cd02f-2573-45e8-a321-5de35dc4c38f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" Jul 23 11:27:44.734757 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.734718 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8q59n" event={"ID":"1ca94b96-042b-4db5-af4d-9855aec04c28","Type":"ContainerStarted","Data":"f64445cf74b0b87d4e1fa7f51135325ee29f10d11009d063c1c3fee1462f2a10"} Jul 23 11:27:44.767983 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:44.767931 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" Jul 23 11:27:47.966529 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:47.966432 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt"] Jul 23 11:27:47.981432 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:47.981399 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c8fd7467-c8m8c"] Jul 23 11:27:48.137009 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:48.136972 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4cd02f_2573_45e8_a321_5de35dc4c38f.slice/crio-47011665c210dfb3c6cea95e147deb45ae28e07cc667676b5fc5d2e34c6ba6f1 WatchSource:0}: Error finding container 47011665c210dfb3c6cea95e147deb45ae28e07cc667676b5fc5d2e34c6ba6f1: Status 404 returned error can't find the container with id 47011665c210dfb3c6cea95e147deb45ae28e07cc667676b5fc5d2e34c6ba6f1 Jul 23 11:27:48.137884 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:48.137696 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079a9f7f_2cd8_4fd4_9c10_51b39cbd4de9.slice/crio-f91ed0b62dbc63ae5a5069a926d975e408306360aac13ab21dc29b5097826509 WatchSource:0}: Error finding container f91ed0b62dbc63ae5a5069a926d975e408306360aac13ab21dc29b5097826509: Status 404 returned error can't find the container with id f91ed0b62dbc63ae5a5069a926d975e408306360aac13ab21dc29b5097826509 Jul 23 11:27:48.748200 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:48.747839 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" event={"ID":"0a4cd02f-2573-45e8-a321-5de35dc4c38f","Type":"ContainerStarted","Data":"47011665c210dfb3c6cea95e147deb45ae28e07cc667676b5fc5d2e34c6ba6f1"} Jul 23 11:27:48.749973 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:48.749920 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fvjmm" event={"ID":"83c4d15b-687e-487a-8f65-12945db48eac","Type":"ContainerStarted","Data":"673e450e1998e8058e638529ba5323be4fe802bd80a8a754bdae62b3a3394cd7"} Jul 23 11:27:48.752416 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:48.752378 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8q59n" event={"ID":"1ca94b96-042b-4db5-af4d-9855aec04c28","Type":"ContainerStarted","Data":"34931be0d87e1b691cd0a15ab481b81cff010fdb6622c630bbda8e7a8bcd3b63"} Jul 23 11:27:48.752620 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:48.752606 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:48.754520 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:48.754487 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q8j9" event={"ID":"8580b584-f0d6-439a-b6d7-7e6d29ae82be","Type":"ContainerStarted","Data":"f77eb18aa014508de37905045f14c4c4cd4a9d1c7a3a0dc27eb486c9fe498062"} Jul 23 11:27:48.756482 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:48.756456 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c8fd7467-c8m8c" event={"ID":"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9","Type":"ContainerStarted","Data":"f91ed0b62dbc63ae5a5069a926d975e408306360aac13ab21dc29b5097826509"} Jul 23 11:27:48.785968 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:48.785896 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fvjmm" podStartSLOduration=1.9496795919999999 podStartE2EDuration="7.785869224s" totalImagesPullingTime="5.836189632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:48.768083682 +0000 UTC m=+60.785970329" watchObservedRunningTime="2026-07-23 11:27:48.785869224 +0000 UTC m=+60.803755844" Jul 23 11:27:48.786459 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:48.786415 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8q59n" podStartSLOduration=5.691567758 podStartE2EDuration="7.786401835s" totalImagesPullingTime="2.094834077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:48.785415995 +0000 UTC m=+60.803302655" watchObservedRunningTime="2026-07-23 11:27:48.786401835 +0000 UTC m=+60.804288472" Jul 23 11:27:49.660668 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:49.660638 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnjnv" Jul 23 11:27:49.765298 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:49.765113 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" event={"ID":"0a4cd02f-2573-45e8-a321-5de35dc4c38f","Type":"ContainerStarted","Data":"d0cc54cc64bf9bf991bad2ae5144cf9d6db6fa731ed0d60f393ab7dac09a61ae"} Jul 23 11:27:49.765298 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:49.765175 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" Jul 23 11:27:49.771731 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:49.771694 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" Jul 23 11:27:49.801739 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:49.800688 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g4lwt" podStartSLOduration=4.724367987 podStartE2EDuration="5.800664367s" totalImagesPullingTime="1.07629638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:49.782716143 +0000 UTC m=+61.800602779" watchObservedRunningTime="2026-07-23 11:27:49.800664367 +0000 UTC m=+61.818550987" Jul 23 11:27:50.522882 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.522847 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5"] Jul 23 11:27:50.524906 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.524873 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.527281 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.527653 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.527811 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a925b66d-6ea3-4a1b-9528-ecda0d251c92-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.527842 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.527853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a925b66d-6ea3-4a1b-9528-ecda0d251c92-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.527909 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a925b66d-6ea3-4a1b-9528-ecda0d251c92-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.527974 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfdp\" (UniqueName: \"kubernetes.io/projected/a925b66d-6ea3-4a1b-9528-ecda0d251c92-kube-api-access-5vfdp\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.528187 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.528434 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-v8lcx\"" Jul 23 11:27:50.528666 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.528473 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 11:27:50.534491 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.534409 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5"] Jul 23 11:27:50.628412 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.628372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a925b66d-6ea3-4a1b-9528-ecda0d251c92-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.628631 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.628431 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a925b66d-6ea3-4a1b-9528-ecda0d251c92-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.628631 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.628486 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a925b66d-6ea3-4a1b-9528-ecda0d251c92-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.628926 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.628900 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfdp\" (UniqueName: \"kubernetes.io/projected/a925b66d-6ea3-4a1b-9528-ecda0d251c92-kube-api-access-5vfdp\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.629276 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.629253 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a925b66d-6ea3-4a1b-9528-ecda0d251c92-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.631713 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.631675 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a925b66d-6ea3-4a1b-9528-ecda0d251c92-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.634031 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.633968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a925b66d-6ea3-4a1b-9528-ecda0d251c92-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.638953 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.638925 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfdp\" (UniqueName: \"kubernetes.io/projected/a925b66d-6ea3-4a1b-9528-ecda0d251c92-kube-api-access-5vfdp\") pod \"prometheus-operator-5945b5cfd-6z4q5\" (UID: \"a925b66d-6ea3-4a1b-9528-ecda0d251c92\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:50.838304 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:50.838261 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" Jul 23 11:27:53.251754 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.251710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:53.252216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.251760 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:53.252216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.251819 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:53.254362 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.254336 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 11:27:53.254495 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.254339 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 11:27:53.255153 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.255133 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 11:27:53.265480 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.265451 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 11:27:53.265632 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.265511 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/110693d2-151e-459c-8789-094e101446fc-original-pull-secret\") pod \"global-pull-secret-syncer-4bqk7\" (UID: \"110693d2-151e-459c-8789-094e101446fc\") " pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:53.265867 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.265846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b8e7c6-8c74-4b67-bcb3-fa61744d2214-metrics-certs\") pod \"network-metrics-daemon-zgbls\" (UID: \"97b8e7c6-8c74-4b67-bcb3-fa61744d2214\") " pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:53.275975 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.275947 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n8ss\" (UniqueName: \"kubernetes.io/projected/8b17bb8b-f1a8-468e-b1fb-64f534c30019-kube-api-access-9n8ss\") pod \"network-check-target-tn5xd\" (UID: \"8b17bb8b-f1a8-468e-b1fb-64f534c30019\") " pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:53.323595 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.323196 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6857cdc795-ctzdp"] Jul 23 11:27:53.325147 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.325123 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.333729 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.333699 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 11:27:53.336992 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.336429 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6857cdc795-ctzdp"] Jul 23 11:27:53.353131 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.353097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-service-ca\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.353331 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.353161 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-oauth-serving-cert\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.353331 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.353187 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-trusted-ca-bundle\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.353331 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.353225 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-config\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.353331 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.353266 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-serving-cert\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.353331 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.353299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lt4b\" (UniqueName: \"kubernetes.io/projected/e4eeea15-a41e-4b4f-8773-04f43cb61a86-kube-api-access-4lt4b\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.353331 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.353321 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-oauth-config\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.383102 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.382847 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5"] Jul 23 11:27:53.387550 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:53.387520 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda925b66d_6ea3_4a1b_9528_ecda0d251c92.slice/crio-160ff8647cbdc651109420c641df87c21f9b8e760bccd21c76976258d187916b WatchSource:0}: Error finding container 160ff8647cbdc651109420c641df87c21f9b8e760bccd21c76976258d187916b: Status 404 returned error can't find the container with id 160ff8647cbdc651109420c641df87c21f9b8e760bccd21c76976258d187916b Jul 23 11:27:53.453738 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.453640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lt4b\" (UniqueName: \"kubernetes.io/projected/e4eeea15-a41e-4b4f-8773-04f43cb61a86-kube-api-access-4lt4b\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.453738 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.453683 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-oauth-config\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.453738 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.453722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-service-ca\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.453738 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.453740 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-oauth-serving-cert\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.454022 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.453757 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-trusted-ca-bundle\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.454022 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.453794 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-config\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.454022 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.453872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-serving-cert\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.454712 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.454680 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-oauth-serving-cert\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.454712 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.454680 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-service-ca\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.454979 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.454954 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-trusted-ca-bundle\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.455039 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.455000 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-config\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.456519 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.456491 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-serving-cert\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.456641 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.456533 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-oauth-config\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.465294 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.465259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lt4b\" (UniqueName: \"kubernetes.io/projected/e4eeea15-a41e-4b4f-8773-04f43cb61a86-kube-api-access-4lt4b\") pod \"console-6857cdc795-ctzdp\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.507731 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.507694 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8j6tk\"" Jul 23 11:27:53.511641 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.511616 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqk7" Jul 23 11:27:53.515675 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.515637 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:27:53.517964 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.517942 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zh6tm\"" Jul 23 11:27:53.526746 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.526715 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zgbls" Jul 23 11:27:53.642332 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.642296 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:27:53.696674 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.696623 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tn5xd"] Jul 23 11:27:53.698645 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.698608 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zgbls"] Jul 23 11:27:53.701392 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:53.701359 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b17bb8b_f1a8_468e_b1fb_64f534c30019.slice/crio-adbde40f564e15b2d3a8c24bd45f56b25016fcbfecdde3ca653fa12e19a9b6f9 WatchSource:0}: Error finding container adbde40f564e15b2d3a8c24bd45f56b25016fcbfecdde3ca653fa12e19a9b6f9: Status 404 returned error can't find the container with id adbde40f564e15b2d3a8c24bd45f56b25016fcbfecdde3ca653fa12e19a9b6f9 Jul 23 11:27:53.701652 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:53.701628 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b8e7c6_8c74_4b67_bcb3_fa61744d2214.slice/crio-cbb9af314fc5dc0336b01b077cdae77adee390815aa509b7f020adda21c3f65f WatchSource:0}: Error finding container cbb9af314fc5dc0336b01b077cdae77adee390815aa509b7f020adda21c3f65f: Status 404 returned error can't find the container with id cbb9af314fc5dc0336b01b077cdae77adee390815aa509b7f020adda21c3f65f Jul 23 11:27:53.717101 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.716665 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4bqk7"] Jul 23 11:27:53.722781 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:53.722744 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110693d2_151e_459c_8789_094e101446fc.slice/crio-da604ed4c822a088560dce0c2e10847029bc9f851e8a4707ab68d473f70ed635 WatchSource:0}: Error finding container da604ed4c822a088560dce0c2e10847029bc9f851e8a4707ab68d473f70ed635: Status 404 returned error can't find the container with id da604ed4c822a088560dce0c2e10847029bc9f851e8a4707ab68d473f70ed635 Jul 23 11:27:53.780771 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.780735 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tn5xd" event={"ID":"8b17bb8b-f1a8-468e-b1fb-64f534c30019","Type":"ContainerStarted","Data":"adbde40f564e15b2d3a8c24bd45f56b25016fcbfecdde3ca653fa12e19a9b6f9"} Jul 23 11:27:53.781819 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.781789 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4bqk7" event={"ID":"110693d2-151e-459c-8789-094e101446fc","Type":"ContainerStarted","Data":"da604ed4c822a088560dce0c2e10847029bc9f851e8a4707ab68d473f70ed635"} Jul 23 11:27:53.783607 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.783561 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q8j9" event={"ID":"8580b584-f0d6-439a-b6d7-7e6d29ae82be","Type":"ContainerStarted","Data":"8795866d820ed806b9b66010d85ff450cb422e1d0f96aaf605993c01f01515f9"} Jul 23 11:27:53.784871 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.784842 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" event={"ID":"a925b66d-6ea3-4a1b-9528-ecda0d251c92","Type":"ContainerStarted","Data":"160ff8647cbdc651109420c641df87c21f9b8e760bccd21c76976258d187916b"} Jul 23 11:27:53.786432 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.786406 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c8fd7467-c8m8c" event={"ID":"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9","Type":"ContainerStarted","Data":"2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a"} Jul 23 11:27:53.787564 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.787542 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zgbls" event={"ID":"97b8e7c6-8c74-4b67-bcb3-fa61744d2214","Type":"ContainerStarted","Data":"cbb9af314fc5dc0336b01b077cdae77adee390815aa509b7f020adda21c3f65f"} Jul 23 11:27:53.797289 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.797257 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6857cdc795-ctzdp"] Jul 23 11:27:53.800217 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:27:53.800182 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4eeea15_a41e_4b4f_8773_04f43cb61a86.slice/crio-3a955f621efc864aefa69907d4c514d54925548ab3a552bfd855c2d9a3461e8c WatchSource:0}: Error finding container 3a955f621efc864aefa69907d4c514d54925548ab3a552bfd855c2d9a3461e8c: Status 404 returned error can't find the container with id 3a955f621efc864aefa69907d4c514d54925548ab3a552bfd855c2d9a3461e8c Jul 23 11:27:53.803446 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:53.803400 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78c8fd7467-c8m8c" podStartSLOduration=4.736247225 podStartE2EDuration="9.803379891s" totalImagesPullingTime="5.067132666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:53.802317495 +0000 UTC m=+65.820204145" watchObservedRunningTime="2026-07-23 11:27:53.803379891 +0000 UTC m=+65.821266527" Jul 23 11:27:54.662386 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:54.662350 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:54.662905 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:54.662464 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:27:54.664280 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:54.663912 2556 patch_prober.go:28] interesting pod/console-78c8fd7467-c8m8c container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 23 11:27:54.664280 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:54.663974 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-78c8fd7467-c8m8c" podUID="079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 23 11:27:54.795405 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:54.795355 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6857cdc795-ctzdp" event={"ID":"e4eeea15-a41e-4b4f-8773-04f43cb61a86","Type":"ContainerStarted","Data":"41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5"} Jul 23 11:27:54.795617 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:54.795422 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6857cdc795-ctzdp" event={"ID":"e4eeea15-a41e-4b4f-8773-04f43cb61a86","Type":"ContainerStarted","Data":"3a955f621efc864aefa69907d4c514d54925548ab3a552bfd855c2d9a3461e8c"} Jul 23 11:27:54.814524 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:54.813749 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6857cdc795-ctzdp" podStartSLOduration=1.8137253869999999 podStartE2EDuration="1.813725387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:54.812846449 +0000 UTC m=+66.830733084" watchObservedRunningTime="2026-07-23 11:27:54.813725387 +0000 UTC m=+66.831612023" Jul 23 11:27:55.768208 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:55.768168 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8q59n" Jul 23 11:27:56.807797 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:56.807757 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" event={"ID":"a925b66d-6ea3-4a1b-9528-ecda0d251c92","Type":"ContainerStarted","Data":"394c750c4e30cc733b3e3add72725cf7484900597d40b1892d2312f95bc576b2"} Jul 23 11:27:56.808275 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:56.807829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" event={"ID":"a925b66d-6ea3-4a1b-9528-ecda0d251c92","Type":"ContainerStarted","Data":"868bf2400ea4ba9f09c9fd93cfbb1a23f9f16d50fc6b9298e6caa59c1e5ab572"} Jul 23 11:27:56.810583 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:56.810529 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zgbls" event={"ID":"97b8e7c6-8c74-4b67-bcb3-fa61744d2214","Type":"ContainerStarted","Data":"a516e6a9aa0ac1c0d736b8adb22a520816958f1c1627be669b60ca82b77b5dd2"} Jul 23 11:27:56.810743 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:56.810617 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zgbls" event={"ID":"97b8e7c6-8c74-4b67-bcb3-fa61744d2214","Type":"ContainerStarted","Data":"8fc8510293032ca794a9a76db7779da2a1f2e190fa997479262078ae6f6549a0"} Jul 23 11:27:56.826716 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:56.825975 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6z4q5" podStartSLOduration=4.369697198 podStartE2EDuration="6.825950793s" totalImagesPullingTime="2.456253595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:56.824826471 +0000 UTC m=+68.842713122" watchObservedRunningTime="2026-07-23 11:27:56.825950793 +0000 UTC m=+68.843837430" Jul 23 11:27:56.841427 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:56.841352 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zgbls" podStartSLOduration=65.695575634 podStartE2EDuration="1m7.841331083s" totalImagesPullingTime="2.145755449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:56.841118177 +0000 UTC m=+68.859004813" watchObservedRunningTime="2026-07-23 11:27:56.841331083 +0000 UTC m=+68.859217720" Jul 23 11:27:58.859302 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.859265 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p"] Jul 23 11:27:58.864187 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.864159 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:58.866826 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.866788 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 23 11:27:58.867185 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.867165 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9jtxd\"" Jul 23 11:27:58.867319 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.867306 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 23 11:27:58.876475 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.876441 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p"] Jul 23 11:27:58.899613 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.899558 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7c725"] Jul 23 11:27:58.900642 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.900606 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec47f205-a8a4-4a70-8723-dc826e0b439f-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:58.900790 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.900662 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wjz\" (UniqueName: \"kubernetes.io/projected/ec47f205-a8a4-4a70-8723-dc826e0b439f-kube-api-access-n6wjz\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:58.900790 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.900710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:58.900790 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.900761 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:58.902881 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.902856 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:58.907580 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.907535 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 11:27:58.908264 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.907939 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m86hz\"" Jul 23 11:27:58.908373 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.908332 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 11:27:58.908512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:58.908488 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 11:27:59.002171 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002121 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec47f205-a8a4-4a70-8723-dc826e0b439f-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.002171 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002176 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-tls\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.002415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002210 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-textfile\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.002415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002242 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wjz\" (UniqueName: \"kubernetes.io/projected/ec47f205-a8a4-4a70-8723-dc826e0b439f-kube-api-access-n6wjz\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.002415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002302 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-sys\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.002415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002341 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a937215-dff4-42e8-a0b7-e85d9f070cb8-metrics-client-ca\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.002415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002398 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.002691 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002470 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.002691 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:59.002512 2556 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 23 11:27:59.002691 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002518 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-wtmp\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.002691 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002542 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.002691 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:59.002594 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-tls podName:ec47f205-a8a4-4a70-8723-dc826e0b439f nodeName:}" failed. No retries permitted until 2026-07-23 11:27:59.502553908 +0000 UTC m=+71.520440525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-25m4p" (UID: "ec47f205-a8a4-4a70-8723-dc826e0b439f") : secret "openshift-state-metrics-tls" not found Jul 23 11:27:59.002691 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002628 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgz7x\" (UniqueName: \"kubernetes.io/projected/1a937215-dff4-42e8-a0b7-e85d9f070cb8-kube-api-access-pgz7x\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.002691 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002671 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.003038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002702 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-root\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.003038 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.002982 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec47f205-a8a4-4a70-8723-dc826e0b439f-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.005639 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.005610 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.013496 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.013464 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wjz\" (UniqueName: \"kubernetes.io/projected/ec47f205-a8a4-4a70-8723-dc826e0b439f-kube-api-access-n6wjz\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.104072 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104020 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-root\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104072 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104080 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-tls\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104112 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-textfile\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104140 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-sys\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104165 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a937215-dff4-42e8-a0b7-e85d9f070cb8-metrics-client-ca\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-root\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104289 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-wtmp\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104326 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgz7x\" (UniqueName: \"kubernetes.io/projected/1a937215-dff4-42e8-a0b7-e85d9f070cb8-kube-api-access-pgz7x\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104687 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104390 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104687 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104407 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-sys\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104687 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104493 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-textfile\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104687 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.104556 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-wtmp\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.104687 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:59.104614 2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 11:27:59.104900 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:27:59.104779 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-tls podName:1a937215-dff4-42e8-a0b7-e85d9f070cb8 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:59.604760049 +0000 UTC m=+71.622646661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-tls") pod "node-exporter-7c725" (UID: "1a937215-dff4-42e8-a0b7-e85d9f070cb8") : secret "node-exporter-tls" not found Jul 23 11:27:59.105544 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.105513 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a937215-dff4-42e8-a0b7-e85d9f070cb8-metrics-client-ca\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.105686 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.105513 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.107703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.107672 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.116156 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.116077 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgz7x\" (UniqueName: \"kubernetes.io/projected/1a937215-dff4-42e8-a0b7-e85d9f070cb8-kube-api-access-pgz7x\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.507639 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.507548 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.510451 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.510417 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec47f205-a8a4-4a70-8723-dc826e0b439f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-25m4p\" (UID: \"ec47f205-a8a4-4a70-8723-dc826e0b439f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.608415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.608372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-tls\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.611205 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.611180 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a937215-dff4-42e8-a0b7-e85d9f070cb8-node-exporter-tls\") pod \"node-exporter-7c725\" (UID: \"1a937215-dff4-42e8-a0b7-e85d9f070cb8\") " pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:27:59.777479 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.777368 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" Jul 23 11:27:59.813875 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:27:59.813840 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7c725" Jul 23 11:28:00.542495 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:28:00.542467 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a937215_dff4_42e8_a0b7_e85d9f070cb8.slice/crio-4e3a488513b0712cc2a7004b24cbc4ecc79196218453ea5db0f613ce346de0a1 WatchSource:0}: Error finding container 4e3a488513b0712cc2a7004b24cbc4ecc79196218453ea5db0f613ce346de0a1: Status 404 returned error can't find the container with id 4e3a488513b0712cc2a7004b24cbc4ecc79196218453ea5db0f613ce346de0a1 Jul 23 11:28:00.695359 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.695303 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p"] Jul 23 11:28:00.699317 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:28:00.699266 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec47f205_a8a4_4a70_8723_dc826e0b439f.slice/crio-789c53cc0dab3c6dcc48a1a644489fd7ac7afa8f2eb9924323b0f0afb2dd2add WatchSource:0}: Error finding container 789c53cc0dab3c6dcc48a1a644489fd7ac7afa8f2eb9924323b0f0afb2dd2add: Status 404 returned error can't find the container with id 789c53cc0dab3c6dcc48a1a644489fd7ac7afa8f2eb9924323b0f0afb2dd2add Jul 23 11:28:00.837300 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.837075 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q8j9" event={"ID":"8580b584-f0d6-439a-b6d7-7e6d29ae82be","Type":"ContainerStarted","Data":"a13e5147a93a5e1c02dbe7b86639f80bacf36573ecfb1cd5157e9175aaa5e096"} Jul 23 11:28:00.839464 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.839429 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" event={"ID":"ec47f205-a8a4-4a70-8723-dc826e0b439f","Type":"ContainerStarted","Data":"5590b7d8241f0b5c965e4de6d8f244b9f4826235bc14d63ed8c89e390e18e2d2"} Jul 23 11:28:00.839541 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.839498 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" event={"ID":"ec47f205-a8a4-4a70-8723-dc826e0b439f","Type":"ContainerStarted","Data":"789c53cc0dab3c6dcc48a1a644489fd7ac7afa8f2eb9924323b0f0afb2dd2add"} Jul 23 11:28:00.840968 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.840895 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7c725" event={"ID":"1a937215-dff4-42e8-a0b7-e85d9f070cb8","Type":"ContainerStarted","Data":"4e3a488513b0712cc2a7004b24cbc4ecc79196218453ea5db0f613ce346de0a1"} Jul 23 11:28:00.843438 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.842796 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tn5xd" event={"ID":"8b17bb8b-f1a8-468e-b1fb-64f534c30019","Type":"ContainerStarted","Data":"157e0a6fef1400052cc372a86ddd2f7e87fbd4a03073991c302aa3668d682949"} Jul 23 11:28:00.843438 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.843151 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:28:00.847076 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.846315 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4bqk7" event={"ID":"110693d2-151e-459c-8789-094e101446fc","Type":"ContainerStarted","Data":"530a2fd7edcc93cd95ee17f0194d930a0ef4899d2b5deb05650b31e4d559b1fd"} Jul 23 11:28:00.857537 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.857479 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4q8j9" podStartSLOduration=1.463759676 podStartE2EDuration="19.857459028s" totalImagesPullingTime="18.393699352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:27:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:00.855474239 +0000 UTC m=+72.873360905" watchObservedRunningTime="2026-07-23 11:28:00.857459028 +0000 UTC m=+72.875345662" Jul 23 11:28:00.873405 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.873350 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4bqk7" podStartSLOduration=65.049796906 podStartE2EDuration="1m11.873333458s" totalImagesPullingTime="6.823536552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:00.872128904 +0000 UTC m=+72.890015561" watchObservedRunningTime="2026-07-23 11:28:00.873333458 +0000 UTC m=+72.891220071" Jul 23 11:28:00.891977 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:00.891919 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tn5xd" podStartSLOduration=65.051660514 podStartE2EDuration="1m11.891900418s" totalImagesPullingTime="6.840239904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:00.890835473 +0000 UTC m=+72.908722108" watchObservedRunningTime="2026-07-23 11:28:00.891900418 +0000 UTC m=+72.909787053" Jul 23 11:28:01.853193 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:01.853154 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" event={"ID":"ec47f205-a8a4-4a70-8723-dc826e0b439f","Type":"ContainerStarted","Data":"4b3e360608c3c78b8557d8e77b04b88e87e81b862dd4c12b401afae98eb3c4b8"} Jul 23 11:28:01.857564 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:01.857528 2556 generic.go:356] "Generic (PLEG): container finished" podID="1a937215-dff4-42e8-a0b7-e85d9f070cb8" containerID="9c8ca9961de7bf4bc87c7f8a3dc7d665a17ef187ae1ed40d7faaf39d4cc66115" exitCode=0 Jul 23 11:28:01.857756 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:01.857626 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7c725" event={"ID":"1a937215-dff4-42e8-a0b7-e85d9f070cb8","Type":"ContainerDied","Data":"9c8ca9961de7bf4bc87c7f8a3dc7d665a17ef187ae1ed40d7faaf39d4cc66115"} Jul 23 11:28:02.864296 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:02.864251 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" event={"ID":"ec47f205-a8a4-4a70-8723-dc826e0b439f","Type":"ContainerStarted","Data":"c59790f902a9f74f127a9b7ab43ea0d98059ec1ec2eb7ac1ea4f1a87fc79cb75"} Jul 23 11:28:02.866530 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:02.866505 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7c725" event={"ID":"1a937215-dff4-42e8-a0b7-e85d9f070cb8","Type":"ContainerStarted","Data":"01001a062910d1f52c1dbf75f7a430f84bd278fe1c3a2e595f4da13a06e3beb8"} Jul 23 11:28:02.866619 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:02.866559 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7c725" event={"ID":"1a937215-dff4-42e8-a0b7-e85d9f070cb8","Type":"ContainerStarted","Data":"33bf9a4c155e3984706de66f16a23f34fa3ec013c09d891e27942c2340e82340"} Jul 23 11:28:02.892370 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:02.892289 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-25m4p" podStartSLOduration=3.742213248 podStartE2EDuration="4.892270035s" totalImagesPullingTime="1.150056787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:02.8912812 +0000 UTC m=+74.909167834" watchObservedRunningTime="2026-07-23 11:28:02.892270035 +0000 UTC m=+74.910156706" Jul 23 11:28:03.229631 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.229510 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7c725" podStartSLOduration=4.321038006 podStartE2EDuration="5.229484825s" totalImagesPullingTime="827.180465ms" totalInitContainerRuntime="81.266354ms" isStatefulPod=true podCreationTimestamp="2026-07-23 11:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:02.920985313 +0000 UTC m=+74.938871948" watchObservedRunningTime="2026-07-23 11:28:03.229484825 +0000 UTC m=+75.247371463" Jul 23 11:28:03.230200 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.230183 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx"] Jul 23 11:28:03.233625 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.233594 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.240858 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.240827 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 23 11:28:03.240858 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.240848 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 23 11:28:03.241228 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.241210 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fpcscqhmupbqo\"" Jul 23 11:28:03.241308 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.241255 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 23 11:28:03.241634 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.241617 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 11:28:03.243081 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.243066 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lfnhk\"" Jul 23 11:28:03.256808 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.256772 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx"] Jul 23 11:28:03.342660 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.342624 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-secret-metrics-server-tls\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.342660 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.342665 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6668\" (UniqueName: \"kubernetes.io/projected/9dd9983c-a018-43dc-9cae-2466adbfe13d-kube-api-access-d6668\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.342903 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.342691 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-secret-metrics-server-client-certs\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.342903 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.342707 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9dd9983c-a018-43dc-9cae-2466adbfe13d-audit-log\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.342903 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.342723 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-client-ca-bundle\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.342903 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.342788 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9dd9983c-a018-43dc-9cae-2466adbfe13d-metrics-server-audit-profiles\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.342903 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.342834 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd9983c-a018-43dc-9cae-2466adbfe13d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.443589 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.443538 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd9983c-a018-43dc-9cae-2466adbfe13d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.443805 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.443651 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-secret-metrics-server-tls\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.443805 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.443682 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6668\" (UniqueName: \"kubernetes.io/projected/9dd9983c-a018-43dc-9cae-2466adbfe13d-kube-api-access-d6668\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.443805 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.443720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-secret-metrics-server-client-certs\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.443805 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.443742 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9dd9983c-a018-43dc-9cae-2466adbfe13d-audit-log\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.443805 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.443764 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-client-ca-bundle\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.444068 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.443806 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9dd9983c-a018-43dc-9cae-2466adbfe13d-metrics-server-audit-profiles\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.444222 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.444187 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9dd9983c-a018-43dc-9cae-2466adbfe13d-audit-log\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.444432 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.444411 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd9983c-a018-43dc-9cae-2466adbfe13d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.444790 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.444771 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9dd9983c-a018-43dc-9cae-2466adbfe13d-metrics-server-audit-profiles\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.446599 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.446538 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-secret-metrics-server-client-certs\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.446983 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.446961 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-secret-metrics-server-tls\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.447053 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.446991 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9dd9983c-a018-43dc-9cae-2466adbfe13d-client-ca-bundle\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.453208 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.453175 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6668\" (UniqueName: \"kubernetes.io/projected/9dd9983c-a018-43dc-9cae-2466adbfe13d-kube-api-access-d6668\") pod \"metrics-server-77b8b7dfc8-lbrxx\" (UID: \"9dd9983c-a018-43dc-9cae-2466adbfe13d\") " pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.543262 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.543151 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:03.643410 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.643370 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:28:03.643620 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.643444 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:28:03.650140 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.650108 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:28:03.694320 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.694276 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx"] Jul 23 11:28:03.697953 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:28:03.697918 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd9983c_a018_43dc_9cae_2466adbfe13d.slice/crio-96cf598e2b7aed81a194f17ef272790e4c7b63f646d16bc519bac0ab4b5dc6ea WatchSource:0}: Error finding container 96cf598e2b7aed81a194f17ef272790e4c7b63f646d16bc519bac0ab4b5dc6ea: Status 404 returned error can't find the container with id 96cf598e2b7aed81a194f17ef272790e4c7b63f646d16bc519bac0ab4b5dc6ea Jul 23 11:28:03.708393 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.708358 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c8fd7467-c8m8c"] Jul 23 11:28:03.719756 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.719719 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq"] Jul 23 11:28:03.724524 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.724498 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:03.727749 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.727721 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 23 11:28:03.727933 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.727764 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k2znt\"" Jul 23 11:28:03.742111 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.742038 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq"] Jul 23 11:28:03.847483 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.847448 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2407f87a-df88-46b0-aeb8-b23af86a9773-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-wthnq\" (UID: \"2407f87a-df88-46b0-aeb8-b23af86a9773\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:03.847483 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.847483 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftmbd\" (UniqueName: \"kubernetes.io/projected/2407f87a-df88-46b0-aeb8-b23af86a9773-kube-api-access-ftmbd\") pod \"monitoring-plugin-7b7b7bc8bf-wthnq\" (UID: \"2407f87a-df88-46b0-aeb8-b23af86a9773\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:03.874387 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.874346 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" event={"ID":"9dd9983c-a018-43dc-9cae-2466adbfe13d","Type":"ContainerStarted","Data":"96cf598e2b7aed81a194f17ef272790e4c7b63f646d16bc519bac0ab4b5dc6ea"} Jul 23 11:28:03.878792 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.878756 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:28:03.949539 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.949123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2407f87a-df88-46b0-aeb8-b23af86a9773-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-wthnq\" (UID: \"2407f87a-df88-46b0-aeb8-b23af86a9773\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:03.949539 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.949471 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftmbd\" (UniqueName: \"kubernetes.io/projected/2407f87a-df88-46b0-aeb8-b23af86a9773-kube-api-access-ftmbd\") pod \"monitoring-plugin-7b7b7bc8bf-wthnq\" (UID: \"2407f87a-df88-46b0-aeb8-b23af86a9773\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:03.952168 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.952140 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2407f87a-df88-46b0-aeb8-b23af86a9773-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-wthnq\" (UID: \"2407f87a-df88-46b0-aeb8-b23af86a9773\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:03.958516 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:03.958480 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftmbd\" (UniqueName: \"kubernetes.io/projected/2407f87a-df88-46b0-aeb8-b23af86a9773-kube-api-access-ftmbd\") pod \"monitoring-plugin-7b7b7bc8bf-wthnq\" (UID: \"2407f87a-df88-46b0-aeb8-b23af86a9773\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:04.035210 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.035167 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:04.116647 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.116541 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4"] Jul 23 11:28:04.120978 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.120954 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.125123 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.125067 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 23 11:28:04.125233 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.125151 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 23 11:28:04.125856 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.125598 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 23 11:28:04.125856 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.125638 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 23 11:28:04.126020 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.125991 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hp588\"" Jul 23 11:28:04.126189 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.126141 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 23 11:28:04.133050 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.133012 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 23 11:28:04.139792 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.139760 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4"] Jul 23 11:28:04.180045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.180011 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq"] Jul 23 11:28:04.183244 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:28:04.183218 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2407f87a_df88_46b0_aeb8_b23af86a9773.slice/crio-99602d4c58f49af0f9416c50e908aba832596184334d63e295c0e509e6f8ca5e WatchSource:0}: Error finding container 99602d4c58f49af0f9416c50e908aba832596184334d63e295c0e509e6f8ca5e: Status 404 returned error can't find the container with id 99602d4c58f49af0f9416c50e908aba832596184334d63e295c0e509e6f8ca5e Jul 23 11:28:04.252096 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.252052 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-serving-certs-ca-bundle\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.252096 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.252101 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.252314 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.252124 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75st\" (UniqueName: \"kubernetes.io/projected/7d79be3c-e42d-4cad-97f5-c0b88a503494-kube-api-access-c75st\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.252314 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.252145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-secret-telemeter-client\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.252314 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.252163 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-metrics-client-ca\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.252314 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.252180 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.252314 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.252263 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-federate-client-tls\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.252314 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.252303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-telemeter-client-tls\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.354112 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.353198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-serving-certs-ca-bundle\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.354112 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.353269 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.354112 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.353310 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75st\" (UniqueName: \"kubernetes.io/projected/7d79be3c-e42d-4cad-97f5-c0b88a503494-kube-api-access-c75st\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.354112 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.353349 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-secret-telemeter-client\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.354112 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.353391 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-metrics-client-ca\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.354112 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.353423 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.354112 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.353492 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-federate-client-tls\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.354112 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.353529 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-telemeter-client-tls\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.355801 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.355767 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-serving-certs-ca-bundle\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.356699 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.356670 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.357302 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.357272 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d79be3c-e42d-4cad-97f5-c0b88a503494-metrics-client-ca\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.359233 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.359199 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-telemeter-client-tls\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.361121 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.361094 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-secret-telemeter-client\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.361267 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.361094 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.361267 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.361158 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d79be3c-e42d-4cad-97f5-c0b88a503494-federate-client-tls\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.366967 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.366881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75st\" (UniqueName: \"kubernetes.io/projected/7d79be3c-e42d-4cad-97f5-c0b88a503494-kube-api-access-c75st\") pod \"telemeter-client-6c9ffd4bcc-5cvz4\" (UID: \"7d79be3c-e42d-4cad-97f5-c0b88a503494\") " pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.434195 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.434149 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" Jul 23 11:28:04.614545 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.614508 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4"] Jul 23 11:28:04.618879 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:28:04.618797 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d79be3c_e42d_4cad_97f5_c0b88a503494.slice/crio-58f9c56ac5ba8875c78d284dfa9ed9e395afe9a564cb5cc16adf220fab657c0d WatchSource:0}: Error finding container 58f9c56ac5ba8875c78d284dfa9ed9e395afe9a564cb5cc16adf220fab657c0d: Status 404 returned error can't find the container with id 58f9c56ac5ba8875c78d284dfa9ed9e395afe9a564cb5cc16adf220fab657c0d Jul 23 11:28:04.881876 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.881768 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" event={"ID":"7d79be3c-e42d-4cad-97f5-c0b88a503494","Type":"ContainerStarted","Data":"58f9c56ac5ba8875c78d284dfa9ed9e395afe9a564cb5cc16adf220fab657c0d"} Jul 23 11:28:04.883405 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:04.883362 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" event={"ID":"2407f87a-df88-46b0-aeb8-b23af86a9773","Type":"ContainerStarted","Data":"99602d4c58f49af0f9416c50e908aba832596184334d63e295c0e509e6f8ca5e"} Jul 23 11:28:05.267391 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.265607 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:28:05.273316 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.273280 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.275515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.275487 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 11:28:05.275713 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.275647 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.276035 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xs4nz\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.276262 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.276494 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.276512 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.276743 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.276753 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.277070 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.277248 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-t0mmqglsvml7\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.277288 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.277250 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.277657 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.277682 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 11:28:05.277751 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.277698 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 11:28:05.281674 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.281651 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 11:28:05.290368 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.290260 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:28:05.365441 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365397 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.365441 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.365753 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365543 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.365753 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365614 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.365753 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365643 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.365753 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365685 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-config\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.365753 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365739 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366035 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365781 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-web-config\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366035 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365810 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366035 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365851 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-config-out\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366035 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366035 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366035 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365955 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366035 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.365984 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366379 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.366037 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366379 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.366066 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366379 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.366105 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366379 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.366149 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqzpg\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-kube-api-access-cqzpg\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366379 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.366176 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.366379 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.366210 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.467598 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467537 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.467802 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467610 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.467802 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.467802 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467668 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqzpg\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-kube-api-access-cqzpg\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.467802 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467697 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.467802 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467757 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.467802 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467789 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467837 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467865 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467887 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467921 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-config\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467946 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.467971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-web-config\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.468000 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.468039 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-config-out\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468080 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.468061 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468523 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.468093 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468523 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.468126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.468523 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.468163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.469759 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.469318 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.469759 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.469729 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.470046 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.470016 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.470533 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.470491 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.471127 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.471099 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.471259 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.471238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.471330 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.471285 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.473842 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.473751 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.474221 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.474193 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-config\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.475216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.474694 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-config-out\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.475216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.475031 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.475216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.475158 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.475216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.475174 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.476870 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.476159 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.476870 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.476823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.478410 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.477482 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.478410 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.477923 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-web-config\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.478410 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.478374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.479505 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.479481 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqzpg\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-kube-api-access-cqzpg\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.480622 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.480558 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:05.589915 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:05.589869 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:06.029971 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:06.029851 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:28:06.034448 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:28:06.034410 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270f23ba_2a79_41df_a6d1_d16953f97028.slice/crio-044a0cfb912b773b1192962b28f68b8444c8945440ee82f73f747558ab41c3bf WatchSource:0}: Error finding container 044a0cfb912b773b1192962b28f68b8444c8945440ee82f73f747558ab41c3bf: Status 404 returned error can't find the container with id 044a0cfb912b773b1192962b28f68b8444c8945440ee82f73f747558ab41c3bf Jul 23 11:28:06.896143 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:06.896106 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" event={"ID":"2407f87a-df88-46b0-aeb8-b23af86a9773","Type":"ContainerStarted","Data":"d4945857894d6bf40fc52d88cd113300a68d92e83b86238f6f14b5828d890081"} Jul 23 11:28:06.896484 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:06.896463 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:06.899410 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:06.899376 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" event={"ID":"9dd9983c-a018-43dc-9cae-2466adbfe13d","Type":"ContainerStarted","Data":"207c019d7a3493642226ca3663d2cbe1ca84156984cf21aa216bf54722adee49"} Jul 23 11:28:06.901097 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:06.900988 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerStarted","Data":"044a0cfb912b773b1192962b28f68b8444c8945440ee82f73f747558ab41c3bf"} Jul 23 11:28:06.903804 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:06.903777 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" Jul 23 11:28:06.920205 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:06.920153 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wthnq" podStartSLOduration=2.256166009 podStartE2EDuration="3.920130451s" totalImagesPullingTime="1.663964442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:06.917016323 +0000 UTC m=+78.934902959" watchObservedRunningTime="2026-07-23 11:28:06.920130451 +0000 UTC m=+78.938017086" Jul 23 11:28:06.940733 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:06.940678 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" podStartSLOduration=1.796618669 podStartE2EDuration="3.940657284s" totalImagesPullingTime="2.144038615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:06.938654117 +0000 UTC m=+78.956540752" watchObservedRunningTime="2026-07-23 11:28:06.940657284 +0000 UTC m=+78.958543919" Jul 23 11:28:07.906946 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:07.906910 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerStarted","Data":"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20"} Jul 23 11:28:07.908620 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:07.908589 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" event={"ID":"7d79be3c-e42d-4cad-97f5-c0b88a503494","Type":"ContainerStarted","Data":"6366e8676e482d1c7f5b27f973abc0a3fea0dbc0c96dfe6dac7a7f766817c05a"} Jul 23 11:28:08.913953 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:08.913910 2556 generic.go:356] "Generic (PLEG): container finished" podID="270f23ba-2a79-41df-a6d1-d16953f97028" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" exitCode=0 Jul 23 11:28:08.914436 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:08.914008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerDied","Data":"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20"} Jul 23 11:28:08.916324 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:08.916294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" event={"ID":"7d79be3c-e42d-4cad-97f5-c0b88a503494","Type":"ContainerStarted","Data":"057f248cba4e99508c8c0ad3cb3d96f52d970d1639f78070da7842335da228e0"} Jul 23 11:28:08.916402 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:08.916361 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" event={"ID":"7d79be3c-e42d-4cad-97f5-c0b88a503494","Type":"ContainerStarted","Data":"c397c3addac7f5d311d8021c9a9f752cd249b394969bf9695718451c58646693"} Jul 23 11:28:08.962877 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:08.962817 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c9ffd4bcc-5cvz4" podStartSLOduration=1.7293564 podStartE2EDuration="4.96279992s" totalImagesPullingTime="3.23344352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:28:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:08.961061634 +0000 UTC m=+80.978948277" watchObservedRunningTime="2026-07-23 11:28:08.96279992 +0000 UTC m=+80.980686582" Jul 23 11:28:12.937583 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:12.937526 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerStarted","Data":"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323"} Jul 23 11:28:12.938020 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:12.937627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerStarted","Data":"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd"} Jul 23 11:28:14.953930 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:14.953835 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerStarted","Data":"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07"} Jul 23 11:28:14.953930 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:14.953933 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerStarted","Data":"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c"} Jul 23 11:28:14.954444 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:14.953958 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerStarted","Data":"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce"} Jul 23 11:28:14.954444 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:14.953982 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerStarted","Data":"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50"} Jul 23 11:28:14.986007 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:14.985943 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.934525599 podStartE2EDuration="9.985919041s" totalImagesPullingTime="7.026868588s" totalInitContainerRuntime="24.524854ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:28:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:14.98443533 +0000 UTC m=+87.002321966" watchObservedRunningTime="2026-07-23 11:28:14.985919041 +0000 UTC m=+87.003805678" Jul 23 11:28:15.590700 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:15.590661 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:28:21.164350 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:21.164303 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6857cdc795-ctzdp"] Jul 23 11:28:23.543911 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:23.543870 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:23.544331 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:23.543931 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:28.730927 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:28.730856 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78c8fd7467-c8m8c" podUID="079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" containerName="console" containerID="cri-o://2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a" gracePeriod=15 Jul 23 11:28:29.520858 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.520824 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:28:29.696359 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696250 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-config\") pod \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " Jul 23 11:28:29.696359 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696306 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-service-ca\") pod \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " Jul 23 11:28:29.696359 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696330 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-oauth-config\") pod \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " Jul 23 11:28:29.696658 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696545 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-serving-cert\") pod \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " Jul 23 11:28:29.696658 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696645 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwd9n\" (UniqueName: \"kubernetes.io/projected/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-kube-api-access-qwd9n\") pod \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " Jul 23 11:28:29.696769 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696720 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-oauth-serving-cert\") pod \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\" (UID: \"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9\") " Jul 23 11:28:29.696769 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696739 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-config" (OuterVolumeSpecName: "console-config") pod "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" (UID: "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:29.696769 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696756 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-service-ca" (OuterVolumeSpecName: "service-ca") pod "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" (UID: "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:29.696979 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696951 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:29.696979 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.696974 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:29.697122 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.697099 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" (UID: "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:29.699003 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.698968 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" (UID: "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:28:29.699116 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.699062 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-kube-api-access-qwd9n" (OuterVolumeSpecName: "kube-api-access-qwd9n") pod "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" (UID: "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9"). InnerVolumeSpecName "kube-api-access-qwd9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:28:29.699116 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.699088 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" (UID: "079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:28:29.798256 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.798215 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-oauth-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:29.798256 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.798249 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-console-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:29.798256 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.798258 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwd9n\" (UniqueName: \"kubernetes.io/projected/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-kube-api-access-qwd9n\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:29.798256 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:29.798267 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9-oauth-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:30.016388 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.016278 2556 generic.go:356] "Generic (PLEG): container finished" podID="079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" containerID="2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a" exitCode=0 Jul 23 11:28:30.016388 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.016342 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c8fd7467-c8m8c" Jul 23 11:28:30.016663 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.016377 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c8fd7467-c8m8c" event={"ID":"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9","Type":"ContainerDied","Data":"2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a"} Jul 23 11:28:30.016663 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.016449 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c8fd7467-c8m8c" event={"ID":"079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9","Type":"ContainerDied","Data":"f91ed0b62dbc63ae5a5069a926d975e408306360aac13ab21dc29b5097826509"} Jul 23 11:28:30.016663 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.016478 2556 scope.go:117] "RemoveContainer" containerID="2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a" Jul 23 11:28:30.025765 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.025742 2556 scope.go:117] "RemoveContainer" containerID="2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a" Jul 23 11:28:30.026096 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:28:30.026077 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a\": container with ID starting with 2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a not found: ID does not exist" containerID="2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a" Jul 23 11:28:30.026161 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.026105 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a"} err="failed to get container status \"2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a\": rpc error: code = NotFound desc = could not find container \"2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a\": container with ID starting with 2adeab89b97f424ac15df28be309b58f5a62d2744dd3c9001cbc0d12c49cca3a not found: ID does not exist" Jul 23 11:28:30.037952 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.037919 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c8fd7467-c8m8c"] Jul 23 11:28:30.041256 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.041223 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78c8fd7467-c8m8c"] Jul 23 11:28:30.499227 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:30.499193 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" path="/var/lib/kubelet/pods/079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9/volumes" Jul 23 11:28:31.861398 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:31.861366 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tn5xd" Jul 23 11:28:43.549353 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:43.549316 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:43.553469 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:43.553439 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77b8b7dfc8-lbrxx" Jul 23 11:28:46.185997 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.185924 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6857cdc795-ctzdp" podUID="e4eeea15-a41e-4b4f-8773-04f43cb61a86" containerName="console" containerID="cri-o://41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5" gracePeriod=15 Jul 23 11:28:46.440948 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.440873 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:28:46.524107 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524068 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-serving-cert\") pod \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " Jul 23 11:28:46.524287 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524126 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lt4b\" (UniqueName: \"kubernetes.io/projected/e4eeea15-a41e-4b4f-8773-04f43cb61a86-kube-api-access-4lt4b\") pod \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " Jul 23 11:28:46.524287 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524159 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-oauth-serving-cert\") pod \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " Jul 23 11:28:46.524287 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524196 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-oauth-config\") pod \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " Jul 23 11:28:46.524287 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524244 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-service-ca\") pod \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " Jul 23 11:28:46.524287 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524284 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-config\") pod \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " Jul 23 11:28:46.524549 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524322 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-trusted-ca-bundle\") pod \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\" (UID: \"e4eeea15-a41e-4b4f-8773-04f43cb61a86\") " Jul 23 11:28:46.524752 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524720 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e4eeea15-a41e-4b4f-8773-04f43cb61a86" (UID: "e4eeea15-a41e-4b4f-8773-04f43cb61a86"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:46.524928 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524887 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e4eeea15-a41e-4b4f-8773-04f43cb61a86" (UID: "e4eeea15-a41e-4b4f-8773-04f43cb61a86"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:46.525208 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.524955 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-config" (OuterVolumeSpecName: "console-config") pod "e4eeea15-a41e-4b4f-8773-04f43cb61a86" (UID: "e4eeea15-a41e-4b4f-8773-04f43cb61a86"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:46.525208 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.525125 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-service-ca" (OuterVolumeSpecName: "service-ca") pod "e4eeea15-a41e-4b4f-8773-04f43cb61a86" (UID: "e4eeea15-a41e-4b4f-8773-04f43cb61a86"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:46.526749 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.526717 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4eeea15-a41e-4b4f-8773-04f43cb61a86-kube-api-access-4lt4b" (OuterVolumeSpecName: "kube-api-access-4lt4b") pod "e4eeea15-a41e-4b4f-8773-04f43cb61a86" (UID: "e4eeea15-a41e-4b4f-8773-04f43cb61a86"). InnerVolumeSpecName "kube-api-access-4lt4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:28:46.526867 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.526735 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e4eeea15-a41e-4b4f-8773-04f43cb61a86" (UID: "e4eeea15-a41e-4b4f-8773-04f43cb61a86"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:28:46.526867 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.526792 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e4eeea15-a41e-4b4f-8773-04f43cb61a86" (UID: "e4eeea15-a41e-4b4f-8773-04f43cb61a86"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:28:46.625945 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.625903 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:46.625945 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.625937 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:46.625945 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.625949 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-trusted-ca-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:46.626185 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.625965 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:46.626185 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.625977 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lt4b\" (UniqueName: \"kubernetes.io/projected/e4eeea15-a41e-4b4f-8773-04f43cb61a86-kube-api-access-4lt4b\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:46.626185 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.625985 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4eeea15-a41e-4b4f-8773-04f43cb61a86-oauth-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:46.626185 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:46.625994 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4eeea15-a41e-4b4f-8773-04f43cb61a86-console-oauth-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:28:47.088007 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.087962 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4eeea15-a41e-4b4f-8773-04f43cb61a86" containerID="41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5" exitCode=0 Jul 23 11:28:47.088224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.088021 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6857cdc795-ctzdp" Jul 23 11:28:47.088224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.088041 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6857cdc795-ctzdp" event={"ID":"e4eeea15-a41e-4b4f-8773-04f43cb61a86","Type":"ContainerDied","Data":"41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5"} Jul 23 11:28:47.088224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.088117 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6857cdc795-ctzdp" event={"ID":"e4eeea15-a41e-4b4f-8773-04f43cb61a86","Type":"ContainerDied","Data":"3a955f621efc864aefa69907d4c514d54925548ab3a552bfd855c2d9a3461e8c"} Jul 23 11:28:47.088224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.088166 2556 scope.go:117] "RemoveContainer" containerID="41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5" Jul 23 11:28:47.097602 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.097560 2556 scope.go:117] "RemoveContainer" containerID="41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5" Jul 23 11:28:47.097939 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:28:47.097923 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5\": container with ID starting with 41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5 not found: ID does not exist" containerID="41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5" Jul 23 11:28:47.097993 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.097949 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5"} err="failed to get container status \"41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5\": rpc error: code = NotFound desc = could not find container \"41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5\": container with ID starting with 41d26f90eb9ec14539a38a1c0afa631222b3c8b122fb44bb66f7fe6eed2ebdc5 not found: ID does not exist" Jul 23 11:28:47.108921 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.108845 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6857cdc795-ctzdp"] Jul 23 11:28:47.111241 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:47.111205 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6857cdc795-ctzdp"] Jul 23 11:28:48.499097 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:28:48.499061 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4eeea15-a41e-4b4f-8773-04f43cb61a86" path="/var/lib/kubelet/pods/e4eeea15-a41e-4b4f-8773-04f43cb61a86/volumes" Jul 23 11:29:05.590459 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:05.590413 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:05.611524 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:05.611492 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:06.184472 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:06.184443 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:23.614072 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:23.614029 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:29:23.614958 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:23.614918 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="prometheus" containerID="cri-o://815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" gracePeriod=600 Jul 23 11:29:23.615216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:23.614977 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="thanos-sidecar" containerID="cri-o://308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" gracePeriod=600 Jul 23 11:29:23.615320 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:23.614998 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy-web" containerID="cri-o://33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" gracePeriod=600 Jul 23 11:29:23.615391 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:23.614977 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy" containerID="cri-o://3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" gracePeriod=600 Jul 23 11:29:23.615473 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:23.615039 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="config-reloader" containerID="cri-o://a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" gracePeriod=600 Jul 23 11:29:23.616055 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:23.615052 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" gracePeriod=600 Jul 23 11:29:23.884620 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:23.884590 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.044425 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044384 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-serving-certs-ca-bundle\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044425 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044431 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-config-out\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044457 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqzpg\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-kube-api-access-cqzpg\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044485 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-db\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044512 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-web-config\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044531 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044555 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-tls\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044593 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-tls-assets\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044621 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044645 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-metrics-client-certs\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.044695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044679 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-1\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044716 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-kubelet-serving-ca-bundle\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044746 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-2\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044783 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-grpc-tls\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044807 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-config\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044833 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-0\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044835 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044859 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-thanos-prometheus-http-client-file\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044893 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-kube-rbac-proxy\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044929 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-metrics-client-ca\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.044957 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-trusted-ca-bundle\") pod \"270f23ba-2a79-41df-a6d1-d16953f97028\" (UID: \"270f23ba-2a79-41df-a6d1-d16953f97028\") " Jul 23 11:29:24.045141 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.045122 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:24.045875 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.045399 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.045875 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.045419 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.045875 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.045824 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:29:24.046428 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.046220 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:24.046701 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.046667 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:24.047457 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.047203 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:24.047609 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.047543 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:24.047838 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.047809 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:24.048415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.048359 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.049052 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.049027 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-config" (OuterVolumeSpecName: "config") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.049527 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.049502 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:29:24.049818 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.049686 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-kube-api-access-cqzpg" (OuterVolumeSpecName: "kube-api-access-cqzpg") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "kube-api-access-cqzpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:29:24.049818 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.049768 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-config-out" (OuterVolumeSpecName: "config-out") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:29:24.049962 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.049864 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.050133 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.050102 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.050288 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.050262 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.050370 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.050287 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.050626 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.050607 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.051198 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.051175 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.061097 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.061057 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-web-config" (OuterVolumeSpecName: "web-config") pod "270f23ba-2a79-41df-a6d1-d16953f97028" (UID: "270f23ba-2a79-41df-a6d1-d16953f97028"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:24.146603 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146482 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-config-out\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146603 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146517 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqzpg\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-kube-api-access-cqzpg\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146603 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146529 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-db\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146603 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146539 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-web-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146603 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146559 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146603 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146591 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146603 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146600 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/270f23ba-2a79-41df-a6d1-d16953f97028-tls-assets\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146603 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146608 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146619 2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-metrics-client-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146627 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146635 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146646 2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-grpc-tls\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146654 2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146661 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146670 2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146678 2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/270f23ba-2a79-41df-a6d1-d16953f97028-secret-kube-rbac-proxy\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146686 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-configmap-metrics-client-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.146966 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.146694 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/270f23ba-2a79-41df-a6d1-d16953f97028-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:29:24.246874 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.246839 2556 generic.go:356] "Generic (PLEG): container finished" podID="270f23ba-2a79-41df-a6d1-d16953f97028" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" exitCode=0 Jul 23 11:29:24.246874 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.246867 2556 generic.go:356] "Generic (PLEG): container finished" podID="270f23ba-2a79-41df-a6d1-d16953f97028" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" exitCode=0 Jul 23 11:29:24.246874 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.246876 2556 generic.go:356] "Generic (PLEG): container finished" podID="270f23ba-2a79-41df-a6d1-d16953f97028" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" exitCode=0 Jul 23 11:29:24.246874 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.246886 2556 generic.go:356] "Generic (PLEG): container finished" podID="270f23ba-2a79-41df-a6d1-d16953f97028" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" exitCode=0 Jul 23 11:29:24.247136 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.246894 2556 generic.go:356] "Generic (PLEG): container finished" podID="270f23ba-2a79-41df-a6d1-d16953f97028" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" exitCode=0 Jul 23 11:29:24.247136 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.246902 2556 generic.go:356] "Generic (PLEG): container finished" podID="270f23ba-2a79-41df-a6d1-d16953f97028" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" exitCode=0 Jul 23 11:29:24.247136 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.246939 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerDied","Data":"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07"} Jul 23 11:29:24.247136 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.247011 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.247136 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.247019 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerDied","Data":"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c"} Jul 23 11:29:24.247136 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.247062 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerDied","Data":"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce"} Jul 23 11:29:24.247136 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.247088 2556 scope.go:117] "RemoveContainer" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" Jul 23 11:29:24.247136 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.247102 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerDied","Data":"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50"} Jul 23 11:29:24.247376 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.247145 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerDied","Data":"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323"} Jul 23 11:29:24.247376 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.247189 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerDied","Data":"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd"} Jul 23 11:29:24.247376 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.247230 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"270f23ba-2a79-41df-a6d1-d16953f97028","Type":"ContainerDied","Data":"044a0cfb912b773b1192962b28f68b8444c8945440ee82f73f747558ab41c3bf"} Jul 23 11:29:24.256183 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.256157 2556 scope.go:117] "RemoveContainer" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" Jul 23 11:29:24.264119 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.264095 2556 scope.go:117] "RemoveContainer" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" Jul 23 11:29:24.276676 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.272864 2556 scope.go:117] "RemoveContainer" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" Jul 23 11:29:24.276676 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.276118 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:29:24.280133 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.280102 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:29:24.282346 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.282324 2556 scope.go:117] "RemoveContainer" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" Jul 23 11:29:24.290172 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.290151 2556 scope.go:117] "RemoveContainer" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" Jul 23 11:29:24.300406 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.300353 2556 scope.go:117] "RemoveContainer" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" Jul 23 11:29:24.305562 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305532 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:29:24.305885 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305862 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy-thanos" Jul 23 11:29:24.305885 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305886 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy-thanos" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305903 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="config-reloader" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305912 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="config-reloader" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305923 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy-web" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305931 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy-web" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305941 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4eeea15-a41e-4b4f-8773-04f43cb61a86" containerName="console" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305946 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4eeea15-a41e-4b4f-8773-04f43cb61a86" containerName="console" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305951 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="init-config-reloader" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305956 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="init-config-reloader" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305962 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="prometheus" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305967 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="prometheus" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305975 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" containerName="console" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305982 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" containerName="console" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.305994 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306003 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306011 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="thanos-sidecar" Jul 23 11:29:24.306023 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306016 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="thanos-sidecar" Jul 23 11:29:24.306515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306071 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy" Jul 23 11:29:24.306515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306079 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy-thanos" Jul 23 11:29:24.306515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306085 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4eeea15-a41e-4b4f-8773-04f43cb61a86" containerName="console" Jul 23 11:29:24.306515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306092 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="kube-rbac-proxy-web" Jul 23 11:29:24.306515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306098 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="prometheus" Jul 23 11:29:24.306515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306104 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="config-reloader" Jul 23 11:29:24.306515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306109 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" containerName="thanos-sidecar" Jul 23 11:29:24.306515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.306115 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="079a9f7f-2cd8-4fd4-9c10-51b39cbd4de9" containerName="console" Jul 23 11:29:24.309940 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.309914 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.310925 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.310905 2556 scope.go:117] "RemoveContainer" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" Jul 23 11:29:24.311297 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:29:24.311225 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": container with ID starting with c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07 not found: ID does not exist" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" Jul 23 11:29:24.311356 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.311314 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07"} err="failed to get container status \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": rpc error: code = NotFound desc = could not find container \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": container with ID starting with c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07 not found: ID does not exist" Jul 23 11:29:24.311409 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.311341 2556 scope.go:117] "RemoveContainer" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" Jul 23 11:29:24.311796 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:29:24.311774 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": container with ID starting with 3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c not found: ID does not exist" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" Jul 23 11:29:24.311896 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.311801 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c"} err="failed to get container status \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": rpc error: code = NotFound desc = could not find container \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": container with ID starting with 3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c not found: ID does not exist" Jul 23 11:29:24.311896 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.311817 2556 scope.go:117] "RemoveContainer" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" Jul 23 11:29:24.312102 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:29:24.312080 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": container with ID starting with 33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce not found: ID does not exist" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" Jul 23 11:29:24.312163 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312110 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce"} err="failed to get container status \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": rpc error: code = NotFound desc = could not find container \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": container with ID starting with 33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce not found: ID does not exist" Jul 23 11:29:24.312163 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312130 2556 scope.go:117] "RemoveContainer" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" Jul 23 11:29:24.312311 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312280 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 11:29:24.312311 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312294 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 11:29:24.312486 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:29:24.312465 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": container with ID starting with 308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50 not found: ID does not exist" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" Jul 23 11:29:24.312546 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312498 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50"} err="failed to get container status \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": rpc error: code = NotFound desc = could not find container \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": container with ID starting with 308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50 not found: ID does not exist" Jul 23 11:29:24.312546 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312519 2556 scope.go:117] "RemoveContainer" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" Jul 23 11:29:24.312677 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312656 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 11:29:24.312735 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312657 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-t0mmqglsvml7\"" Jul 23 11:29:24.312735 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312687 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 11:29:24.312833 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312755 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 11:29:24.312884 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312858 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 11:29:24.312956 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.312941 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 11:29:24.313064 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:29:24.313039 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": container with ID starting with a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323 not found: ID does not exist" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" Jul 23 11:29:24.313189 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313071 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323"} err="failed to get container status \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": rpc error: code = NotFound desc = could not find container \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": container with ID starting with a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323 not found: ID does not exist" Jul 23 11:29:24.313189 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313092 2556 scope.go:117] "RemoveContainer" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" Jul 23 11:29:24.313189 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313102 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 11:29:24.313189 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313186 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xs4nz\"" Jul 23 11:29:24.313415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313219 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 11:29:24.313415 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313348 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 11:29:24.313512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313422 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 11:29:24.313619 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:29:24.313592 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": container with ID starting with 815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd not found: ID does not exist" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" Jul 23 11:29:24.313675 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313630 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd"} err="failed to get container status \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": rpc error: code = NotFound desc = could not find container \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": container with ID starting with 815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd not found: ID does not exist" Jul 23 11:29:24.313675 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313653 2556 scope.go:117] "RemoveContainer" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" Jul 23 11:29:24.313818 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313798 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 11:29:24.313966 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:29:24.313947 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": container with ID starting with 57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20 not found: ID does not exist" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" Jul 23 11:29:24.314043 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313976 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20"} err="failed to get container status \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": rpc error: code = NotFound desc = could not find container \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": container with ID starting with 57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20 not found: ID does not exist" Jul 23 11:29:24.314043 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.313997 2556 scope.go:117] "RemoveContainer" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" Jul 23 11:29:24.314305 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.314278 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07"} err="failed to get container status \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": rpc error: code = NotFound desc = could not find container \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": container with ID starting with c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07 not found: ID does not exist" Jul 23 11:29:24.314305 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.314305 2556 scope.go:117] "RemoveContainer" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" Jul 23 11:29:24.314547 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.314519 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c"} err="failed to get container status \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": rpc error: code = NotFound desc = could not find container \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": container with ID starting with 3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c not found: ID does not exist" Jul 23 11:29:24.314650 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.314551 2556 scope.go:117] "RemoveContainer" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" Jul 23 11:29:24.314829 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.314802 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce"} err="failed to get container status \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": rpc error: code = NotFound desc = could not find container \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": container with ID starting with 33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce not found: ID does not exist" Jul 23 11:29:24.314829 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.314829 2556 scope.go:117] "RemoveContainer" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" Jul 23 11:29:24.315119 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.315092 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50"} err="failed to get container status \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": rpc error: code = NotFound desc = could not find container \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": container with ID starting with 308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50 not found: ID does not exist" Jul 23 11:29:24.315178 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.315122 2556 scope.go:117] "RemoveContainer" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" Jul 23 11:29:24.315419 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.315397 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323"} err="failed to get container status \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": rpc error: code = NotFound desc = could not find container \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": container with ID starting with a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323 not found: ID does not exist" Jul 23 11:29:24.315485 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.315423 2556 scope.go:117] "RemoveContainer" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" Jul 23 11:29:24.315728 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.315704 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd"} err="failed to get container status \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": rpc error: code = NotFound desc = could not find container \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": container with ID starting with 815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd not found: ID does not exist" Jul 23 11:29:24.315813 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.315730 2556 scope.go:117] "RemoveContainer" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" Jul 23 11:29:24.315993 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.315974 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20"} err="failed to get container status \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": rpc error: code = NotFound desc = could not find container \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": container with ID starting with 57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20 not found: ID does not exist" Jul 23 11:29:24.316068 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.315996 2556 scope.go:117] "RemoveContainer" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" Jul 23 11:29:24.316288 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.316263 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07"} err="failed to get container status \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": rpc error: code = NotFound desc = could not find container \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": container with ID starting with c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07 not found: ID does not exist" Jul 23 11:29:24.316338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.316292 2556 scope.go:117] "RemoveContainer" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" Jul 23 11:29:24.316551 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.316534 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c"} err="failed to get container status \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": rpc error: code = NotFound desc = could not find container \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": container with ID starting with 3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c not found: ID does not exist" Jul 23 11:29:24.316629 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.316551 2556 scope.go:117] "RemoveContainer" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" Jul 23 11:29:24.316793 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.316769 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce"} err="failed to get container status \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": rpc error: code = NotFound desc = could not find container \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": container with ID starting with 33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce not found: ID does not exist" Jul 23 11:29:24.316845 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.316793 2556 scope.go:117] "RemoveContainer" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" Jul 23 11:29:24.317135 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.317107 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50"} err="failed to get container status \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": rpc error: code = NotFound desc = could not find container \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": container with ID starting with 308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50 not found: ID does not exist" Jul 23 11:29:24.317215 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.317138 2556 scope.go:117] "RemoveContainer" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" Jul 23 11:29:24.317650 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.317621 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323"} err="failed to get container status \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": rpc error: code = NotFound desc = could not find container \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": container with ID starting with a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323 not found: ID does not exist" Jul 23 11:29:24.317650 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.317650 2556 scope.go:117] "RemoveContainer" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" Jul 23 11:29:24.317986 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.317957 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd"} err="failed to get container status \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": rpc error: code = NotFound desc = could not find container \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": container with ID starting with 815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd not found: ID does not exist" Jul 23 11:29:24.317986 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.317977 2556 scope.go:117] "RemoveContainer" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" Jul 23 11:29:24.318129 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.317990 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 11:29:24.318239 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.318218 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20"} err="failed to get container status \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": rpc error: code = NotFound desc = could not find container \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": container with ID starting with 57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20 not found: ID does not exist" Jul 23 11:29:24.318307 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.318241 2556 scope.go:117] "RemoveContainer" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" Jul 23 11:29:24.318847 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.318519 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07"} err="failed to get container status \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": rpc error: code = NotFound desc = could not find container \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": container with ID starting with c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07 not found: ID does not exist" Jul 23 11:29:24.318847 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.318546 2556 scope.go:117] "RemoveContainer" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" Jul 23 11:29:24.319387 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.318900 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c"} err="failed to get container status \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": rpc error: code = NotFound desc = could not find container \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": container with ID starting with 3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c not found: ID does not exist" Jul 23 11:29:24.319387 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.318926 2556 scope.go:117] "RemoveContainer" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" Jul 23 11:29:24.319387 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.319201 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce"} err="failed to get container status \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": rpc error: code = NotFound desc = could not find container \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": container with ID starting with 33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce not found: ID does not exist" Jul 23 11:29:24.319387 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.319224 2556 scope.go:117] "RemoveContainer" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" Jul 23 11:29:24.319795 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.319488 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50"} err="failed to get container status \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": rpc error: code = NotFound desc = could not find container \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": container with ID starting with 308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50 not found: ID does not exist" Jul 23 11:29:24.319795 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.319512 2556 scope.go:117] "RemoveContainer" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" Jul 23 11:29:24.319909 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.319828 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323"} err="failed to get container status \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": rpc error: code = NotFound desc = could not find container \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": container with ID starting with a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323 not found: ID does not exist" Jul 23 11:29:24.319909 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.319851 2556 scope.go:117] "RemoveContainer" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" Jul 23 11:29:24.320115 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320093 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd"} err="failed to get container status \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": rpc error: code = NotFound desc = could not find container \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": container with ID starting with 815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd not found: ID does not exist" Jul 23 11:29:24.320198 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320117 2556 scope.go:117] "RemoveContainer" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" Jul 23 11:29:24.320435 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320408 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20"} err="failed to get container status \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": rpc error: code = NotFound desc = could not find container \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": container with ID starting with 57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20 not found: ID does not exist" Jul 23 11:29:24.320506 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320437 2556 scope.go:117] "RemoveContainer" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" Jul 23 11:29:24.320506 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320451 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 11:29:24.320723 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320700 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07"} err="failed to get container status \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": rpc error: code = NotFound desc = could not find container \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": container with ID starting with c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07 not found: ID does not exist" Jul 23 11:29:24.320785 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320726 2556 scope.go:117] "RemoveContainer" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" Jul 23 11:29:24.320973 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320955 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c"} err="failed to get container status \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": rpc error: code = NotFound desc = could not find container \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": container with ID starting with 3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c not found: ID does not exist" Jul 23 11:29:24.321029 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.320975 2556 scope.go:117] "RemoveContainer" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" Jul 23 11:29:24.321201 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.321172 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce"} err="failed to get container status \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": rpc error: code = NotFound desc = could not find container \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": container with ID starting with 33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce not found: ID does not exist" Jul 23 11:29:24.321201 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.321198 2556 scope.go:117] "RemoveContainer" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" Jul 23 11:29:24.321416 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.321397 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50"} err="failed to get container status \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": rpc error: code = NotFound desc = could not find container \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": container with ID starting with 308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50 not found: ID does not exist" Jul 23 11:29:24.321498 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.321417 2556 scope.go:117] "RemoveContainer" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" Jul 23 11:29:24.321681 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.321659 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323"} err="failed to get container status \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": rpc error: code = NotFound desc = could not find container \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": container with ID starting with a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323 not found: ID does not exist" Jul 23 11:29:24.321681 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.321680 2556 scope.go:117] "RemoveContainer" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" Jul 23 11:29:24.321906 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.321887 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd"} err="failed to get container status \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": rpc error: code = NotFound desc = could not find container \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": container with ID starting with 815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd not found: ID does not exist" Jul 23 11:29:24.321962 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.321907 2556 scope.go:117] "RemoveContainer" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" Jul 23 11:29:24.322123 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322104 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20"} err="failed to get container status \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": rpc error: code = NotFound desc = could not find container \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": container with ID starting with 57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20 not found: ID does not exist" Jul 23 11:29:24.322191 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322123 2556 scope.go:117] "RemoveContainer" containerID="c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07" Jul 23 11:29:24.322322 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322307 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07"} err="failed to get container status \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": rpc error: code = NotFound desc = could not find container \"c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07\": container with ID starting with c5bf5706e94348a998ecb9754d79fd84edd37a64a0636ea64052b223c6da8f07 not found: ID does not exist" Jul 23 11:29:24.322360 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322323 2556 scope.go:117] "RemoveContainer" containerID="3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c" Jul 23 11:29:24.322537 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322520 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c"} err="failed to get container status \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": rpc error: code = NotFound desc = could not find container \"3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c\": container with ID starting with 3d65ceb20834eb91173523e9636081d0ac60c1f798d2b0969a3d584b1bf2d47c not found: ID does not exist" Jul 23 11:29:24.322612 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322538 2556 scope.go:117] "RemoveContainer" containerID="33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce" Jul 23 11:29:24.322777 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322760 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce"} err="failed to get container status \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": rpc error: code = NotFound desc = could not find container \"33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce\": container with ID starting with 33161e54a0d4a1514df9141bfb5a8502eb9344a674bb844185a58e98d7ec97ce not found: ID does not exist" Jul 23 11:29:24.322822 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322777 2556 scope.go:117] "RemoveContainer" containerID="308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50" Jul 23 11:29:24.322975 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322959 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50"} err="failed to get container status \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": rpc error: code = NotFound desc = could not find container \"308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50\": container with ID starting with 308fdba4cc5f03740fca0c8695bd115b645ac42e8fd181f202df458683c3fd50 not found: ID does not exist" Jul 23 11:29:24.323018 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.322976 2556 scope.go:117] "RemoveContainer" containerID="a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323" Jul 23 11:29:24.323163 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.323142 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323"} err="failed to get container status \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": rpc error: code = NotFound desc = could not find container \"a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323\": container with ID starting with a1222a4ff6b533e25bd236425cd67e937df9013750fdb40cd4561d29ebc8b323 not found: ID does not exist" Jul 23 11:29:24.323228 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.323164 2556 scope.go:117] "RemoveContainer" containerID="815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd" Jul 23 11:29:24.323368 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.323347 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd"} err="failed to get container status \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": rpc error: code = NotFound desc = could not find container \"815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd\": container with ID starting with 815027acdc6083367f367e22146f4cb488b1238237bcd06b28931d80766cb5fd not found: ID does not exist" Jul 23 11:29:24.323412 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.323370 2556 scope.go:117] "RemoveContainer" containerID="57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20" Jul 23 11:29:24.323556 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.323540 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20"} err="failed to get container status \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": rpc error: code = NotFound desc = could not find container \"57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20\": container with ID starting with 57ad89267cb48abc10a7c5de02200f16011e5ec8944783363ff02a6258783a20 not found: ID does not exist" Jul 23 11:29:24.328639 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.328605 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:29:24.448892 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.448783 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-config\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.448892 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.448844 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.448892 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.448866 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.448892 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.448901 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.448946 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.448974 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.448991 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk74b\" (UniqueName: \"kubernetes.io/projected/5aaa6bed-6679-427b-805e-bb52ff158a9a-kube-api-access-pk74b\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449036 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449061 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449103 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449151 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-web-config\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449179 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5aaa6bed-6679-427b-805e-bb52ff158a9a-config-out\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449195 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449216 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449238 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449275 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449326 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5aaa6bed-6679-427b-805e-bb52ff158a9a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449347 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.449383 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.449367 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.499495 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.499463 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="270f23ba-2a79-41df-a6d1-d16953f97028" path="/var/lib/kubelet/pods/270f23ba-2a79-41df-a6d1-d16953f97028/volumes" Jul 23 11:29:24.550452 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550415 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550630 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550630 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550478 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550630 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-web-config\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550630 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5aaa6bed-6679-427b-805e-bb52ff158a9a-config-out\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550847 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550645 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550847 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550847 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550700 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550847 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550756 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.550847 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.550793 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5aaa6bed-6679-427b-805e-bb52ff158a9a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551219 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551196 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551488 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551332 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551488 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551332 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551488 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551454 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551683 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551501 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551683 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-config\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551683 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551603 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551683 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551632 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551683 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551663 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551930 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551693 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.551930 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551695 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.552024 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.551957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.552173 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.552148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.552517 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.552492 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.552951 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.552865 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk74b\" (UniqueName: \"kubernetes.io/projected/5aaa6bed-6679-427b-805e-bb52ff158a9a-kube-api-access-pk74b\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.552951 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.552914 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.553675 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.553601 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.553928 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.553871 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.554054 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.553991 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5aaa6bed-6679-427b-805e-bb52ff158a9a-config-out\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.554196 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.554175 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-web-config\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.556048 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.555992 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.556048 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.556011 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.556240 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.556057 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.556526 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.556486 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.556706 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.556685 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.556854 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.556832 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5aaa6bed-6679-427b-805e-bb52ff158a9a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.557046 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.557024 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5aaa6bed-6679-427b-805e-bb52ff158a9a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.557338 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.557316 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-config\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.558013 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.557994 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5aaa6bed-6679-427b-805e-bb52ff158a9a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.562097 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.562068 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk74b\" (UniqueName: \"kubernetes.io/projected/5aaa6bed-6679-427b-805e-bb52ff158a9a-kube-api-access-pk74b\") pod \"prometheus-k8s-0\" (UID: \"5aaa6bed-6679-427b-805e-bb52ff158a9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.623017 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.622975 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:29:24.775862 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:24.775743 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 11:29:24.778372 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:29:24.778338 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aaa6bed_6679_427b_805e_bb52ff158a9a.slice/crio-c5ae28c990c63b49104e6e6e4622ac3096b444970e132a5f37bb86298af696f7 WatchSource:0}: Error finding container c5ae28c990c63b49104e6e6e4622ac3096b444970e132a5f37bb86298af696f7: Status 404 returned error can't find the container with id c5ae28c990c63b49104e6e6e4622ac3096b444970e132a5f37bb86298af696f7 Jul 23 11:29:25.253547 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:25.253508 2556 generic.go:356] "Generic (PLEG): container finished" podID="5aaa6bed-6679-427b-805e-bb52ff158a9a" containerID="bde4d06ce59a31e5869b2ba243e48f53266809398890f5f0d9e5a0b0c15aa05d" exitCode=0 Jul 23 11:29:25.253776 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:25.253556 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5aaa6bed-6679-427b-805e-bb52ff158a9a","Type":"ContainerDied","Data":"bde4d06ce59a31e5869b2ba243e48f53266809398890f5f0d9e5a0b0c15aa05d"} Jul 23 11:29:25.253776 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:25.253655 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5aaa6bed-6679-427b-805e-bb52ff158a9a","Type":"ContainerStarted","Data":"c5ae28c990c63b49104e6e6e4622ac3096b444970e132a5f37bb86298af696f7"} Jul 23 11:29:26.262242 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:26.262204 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5aaa6bed-6679-427b-805e-bb52ff158a9a","Type":"ContainerStarted","Data":"4edead3e440dfec227e428338d0764c0ba39d24155940f83de2a76abdff3a66a"} Jul 23 11:29:26.262681 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:26.262304 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5aaa6bed-6679-427b-805e-bb52ff158a9a","Type":"ContainerStarted","Data":"a3518113a041fea01fd2ec79a9023b8d4af211ae9f5be42757ef728f817ded6d"} Jul 23 11:29:26.262681 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:26.262343 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5aaa6bed-6679-427b-805e-bb52ff158a9a","Type":"ContainerStarted","Data":"f632e5992e3c3e4fe23fe3f8d61a0138f643f56df1226447c256b1d2d256050e"} Jul 23 11:29:26.262681 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:26.262370 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5aaa6bed-6679-427b-805e-bb52ff158a9a","Type":"ContainerStarted","Data":"f1bf34d784ca95de922bf34301d8570a2be44373da560872604e3532957bc63f"} Jul 23 11:29:26.262681 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:26.262394 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5aaa6bed-6679-427b-805e-bb52ff158a9a","Type":"ContainerStarted","Data":"8c272fcf0c4679cd88c32ca3d9c5a46f3d16c26fd165b19df6fcdbc633547b19"} Jul 23 11:29:26.262681 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:26.262416 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5aaa6bed-6679-427b-805e-bb52ff158a9a","Type":"ContainerStarted","Data":"bef9ac4a8bd1d6c7b432d4b342243aee4d8a1617eacc220bb245ed43752a59f5"} Jul 23 11:29:26.291935 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:26.291866 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.270099917 podStartE2EDuration="2.291842769s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.742852ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:29:26.290290639 +0000 UTC m=+158.308177299" watchObservedRunningTime="2026-07-23 11:29:26.291842769 +0000 UTC m=+158.309729405" Jul 23 11:29:29.623723 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:29:29.623677 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:30:15.868390 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.868348 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6hqmf"] Jul 23 11:30:15.870500 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.870479 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:15.872869 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.872846 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 11:30:15.873005 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.872922 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 11:30:15.873631 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.873615 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2gp7n\"" Jul 23 11:30:15.883364 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.883328 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6hqmf"] Jul 23 11:30:15.895727 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.895692 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ztp\" (UniqueName: \"kubernetes.io/projected/64a05d0a-68ce-42c3-84f7-2daab8af911f-kube-api-access-t9ztp\") pod \"cert-manager-webhook-6486ffd997-6hqmf\" (UID: \"64a05d0a-68ce-42c3-84f7-2daab8af911f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:15.895881 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.895751 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64a05d0a-68ce-42c3-84f7-2daab8af911f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6hqmf\" (UID: \"64a05d0a-68ce-42c3-84f7-2daab8af911f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:15.996793 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.996751 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64a05d0a-68ce-42c3-84f7-2daab8af911f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6hqmf\" (UID: \"64a05d0a-68ce-42c3-84f7-2daab8af911f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:15.996975 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:15.996845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ztp\" (UniqueName: \"kubernetes.io/projected/64a05d0a-68ce-42c3-84f7-2daab8af911f-kube-api-access-t9ztp\") pod \"cert-manager-webhook-6486ffd997-6hqmf\" (UID: \"64a05d0a-68ce-42c3-84f7-2daab8af911f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:16.027135 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:16.027093 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64a05d0a-68ce-42c3-84f7-2daab8af911f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6hqmf\" (UID: \"64a05d0a-68ce-42c3-84f7-2daab8af911f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:16.027135 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:16.027106 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ztp\" (UniqueName: \"kubernetes.io/projected/64a05d0a-68ce-42c3-84f7-2daab8af911f-kube-api-access-t9ztp\") pod \"cert-manager-webhook-6486ffd997-6hqmf\" (UID: \"64a05d0a-68ce-42c3-84f7-2daab8af911f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:16.194374 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:16.194271 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:16.331207 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:16.331171 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6hqmf"] Jul 23 11:30:16.334650 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:30:16.334615 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64a05d0a_68ce_42c3_84f7_2daab8af911f.slice/crio-423e62aa875c17b8642bc6731f16db8386e4b21f698ac19d5cabcea6ff000786 WatchSource:0}: Error finding container 423e62aa875c17b8642bc6731f16db8386e4b21f698ac19d5cabcea6ff000786: Status 404 returned error can't find the container with id 423e62aa875c17b8642bc6731f16db8386e4b21f698ac19d5cabcea6ff000786 Jul 23 11:30:16.457527 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:16.457422 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" event={"ID":"64a05d0a-68ce-42c3-84f7-2daab8af911f","Type":"ContainerStarted","Data":"423e62aa875c17b8642bc6731f16db8386e4b21f698ac19d5cabcea6ff000786"} Jul 23 11:30:20.478097 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:20.478055 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" event={"ID":"64a05d0a-68ce-42c3-84f7-2daab8af911f","Type":"ContainerStarted","Data":"850ec3d24a789246ef229ab674d828a204c6861a8ab1557f69ff19319a800e66"} Jul 23 11:30:20.478512 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:20.478125 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:20.496181 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:20.496129 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" podStartSLOduration=2.216164971 podStartE2EDuration="5.496111684s" totalImagesPullingTime="3.279946713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:30:20.49530992 +0000 UTC m=+212.513196556" watchObservedRunningTime="2026-07-23 11:30:20.496111684 +0000 UTC m=+212.513998319" Jul 23 11:30:24.623782 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:24.623737 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:30:24.640915 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:24.640881 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:30:25.516072 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:25.516041 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 11:30:26.485002 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:26.484971 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6hqmf" Jul 23 11:30:50.787961 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.787927 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv"] Jul 23 11:30:50.790745 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.790725 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:50.797737 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.797713 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 11:30:50.798066 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.798046 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 11:30:50.798172 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.798074 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 11:30:50.798419 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.798401 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5hr4d\"" Jul 23 11:30:50.799107 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.799091 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 11:30:50.844225 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.844183 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv"] Jul 23 11:30:50.914026 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.913988 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6425183-0989-4a79-913f-94573d32fd36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:50.914271 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.914039 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6425183-0989-4a79-913f-94573d32fd36-manifests\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:50.914271 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.914093 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6425183-0989-4a79-913f-94573d32fd36-webhook-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:50.914271 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.914169 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6425183-0989-4a79-913f-94573d32fd36-tmp\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:50.914271 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:50.914222 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8nw\" (UniqueName: \"kubernetes.io/projected/d6425183-0989-4a79-913f-94573d32fd36-kube-api-access-hx8nw\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.015471 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.015427 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6425183-0989-4a79-913f-94573d32fd36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.015695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.015481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6425183-0989-4a79-913f-94573d32fd36-manifests\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.015695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.015506 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6425183-0989-4a79-913f-94573d32fd36-webhook-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.015695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.015534 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6425183-0989-4a79-913f-94573d32fd36-tmp\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.015695 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.015615 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8nw\" (UniqueName: \"kubernetes.io/projected/d6425183-0989-4a79-913f-94573d32fd36-kube-api-access-hx8nw\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.016021 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.015992 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6425183-0989-4a79-913f-94573d32fd36-manifests\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.018093 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.018062 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6425183-0989-4a79-913f-94573d32fd36-tmp\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.018461 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.018438 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6425183-0989-4a79-913f-94573d32fd36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.018550 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.018447 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6425183-0989-4a79-913f-94573d32fd36-webhook-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.025195 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.025167 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8nw\" (UniqueName: \"kubernetes.io/projected/d6425183-0989-4a79-913f-94573d32fd36-kube-api-access-hx8nw\") pod \"opendatahub-operator-controller-manager-674bfcc66c-xqjnv\" (UID: \"d6425183-0989-4a79-913f-94573d32fd36\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.101254 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.101215 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:51.252974 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.252935 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv"] Jul 23 11:30:51.257213 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:30:51.257181 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6425183_0989_4a79_913f_94573d32fd36.slice/crio-390e7f35e99a47c165d8918e4d8cc1df6644185a1ddf50b38a0487ab4d9989bc WatchSource:0}: Error finding container 390e7f35e99a47c165d8918e4d8cc1df6644185a1ddf50b38a0487ab4d9989bc: Status 404 returned error can't find the container with id 390e7f35e99a47c165d8918e4d8cc1df6644185a1ddf50b38a0487ab4d9989bc Jul 23 11:30:51.597804 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:51.597761 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" event={"ID":"d6425183-0989-4a79-913f-94573d32fd36","Type":"ContainerStarted","Data":"390e7f35e99a47c165d8918e4d8cc1df6644185a1ddf50b38a0487ab4d9989bc"} Jul 23 11:30:54.612263 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:54.612226 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" event={"ID":"d6425183-0989-4a79-913f-94573d32fd36","Type":"ContainerStarted","Data":"df0eb8889ba23ea392e668dd094b9fb976b9ee8703cdaf2d8281651efa4e3c04"} Jul 23 11:30:55.618073 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:55.617995 2556 generic.go:356] "Generic (PLEG): container finished" podID="d6425183-0989-4a79-913f-94573d32fd36" containerID="df0eb8889ba23ea392e668dd094b9fb976b9ee8703cdaf2d8281651efa4e3c04" exitCode=0 Jul 23 11:30:55.618073 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:55.618054 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" event={"ID":"d6425183-0989-4a79-913f-94573d32fd36","Type":"ContainerDied","Data":"df0eb8889ba23ea392e668dd094b9fb976b9ee8703cdaf2d8281651efa4e3c04"} Jul 23 11:30:57.628654 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:57.628612 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" event={"ID":"d6425183-0989-4a79-913f-94573d32fd36","Type":"ContainerStarted","Data":"74a190b9360175728c89c1f5c94ee5fa0d6ca7629aa6a02461e6f304a41386f1"} Jul 23 11:30:57.629065 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:57.628759 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:30:57.653830 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:30:57.653777 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" podStartSLOduration=2.625700603 podStartE2EDuration="7.65375782s" totalImagesPullingTime="4.008692348s" totalInitContainerRuntime="1.019364869s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:30:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:30:57.651506748 +0000 UTC m=+249.669393432" watchObservedRunningTime="2026-07-23 11:30:57.65375782 +0000 UTC m=+249.671644454" Jul 23 11:31:08.635765 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:08.635689 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-xqjnv" Jul 23 11:31:11.005010 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.004971 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p"] Jul 23 11:31:11.008373 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.008346 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.011862 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.011838 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 11:31:11.011862 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.011845 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 11:31:11.012794 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.012770 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-66qgl\"" Jul 23 11:31:11.012980 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.012964 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 11:31:11.013076 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.013055 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 11:31:11.024814 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.024777 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p"] Jul 23 11:31:11.077754 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.077709 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83f6ed15-a9de-4496-bb25-3064208777f0-tmp\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.077928 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.077769 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f6ed15-a9de-4496-bb25-3064208777f0-tls-certs\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.077928 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.077831 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5s8t\" (UniqueName: \"kubernetes.io/projected/83f6ed15-a9de-4496-bb25-3064208777f0-kube-api-access-p5s8t\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.179111 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.179073 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83f6ed15-a9de-4496-bb25-3064208777f0-tmp\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.179300 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.179138 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f6ed15-a9de-4496-bb25-3064208777f0-tls-certs\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.179300 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.179164 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5s8t\" (UniqueName: \"kubernetes.io/projected/83f6ed15-a9de-4496-bb25-3064208777f0-kube-api-access-p5s8t\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.182021 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.181988 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83f6ed15-a9de-4496-bb25-3064208777f0-tmp\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.182165 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.182144 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f6ed15-a9de-4496-bb25-3064208777f0-tls-certs\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.198403 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.198364 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5s8t\" (UniqueName: \"kubernetes.io/projected/83f6ed15-a9de-4496-bb25-3064208777f0-kube-api-access-p5s8t\") pod \"kube-auth-proxy-766d5cd5c5-25z6p\" (UID: \"83f6ed15-a9de-4496-bb25-3064208777f0\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.319862 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.319768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" Jul 23 11:31:11.466681 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.466645 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p"] Jul 23 11:31:11.470350 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:31:11.470318 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f6ed15_a9de_4496_bb25_3064208777f0.slice/crio-adf4ac6563653e898ab1d9566a2644bbe733767d5b94586fd8270b1d0267df86 WatchSource:0}: Error finding container adf4ac6563653e898ab1d9566a2644bbe733767d5b94586fd8270b1d0267df86: Status 404 returned error can't find the container with id adf4ac6563653e898ab1d9566a2644bbe733767d5b94586fd8270b1d0267df86 Jul 23 11:31:11.693874 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:11.693837 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" event={"ID":"83f6ed15-a9de-4496-bb25-3064208777f0","Type":"ContainerStarted","Data":"adf4ac6563653e898ab1d9566a2644bbe733767d5b94586fd8270b1d0267df86"} Jul 23 11:31:14.708244 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:14.708208 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" event={"ID":"83f6ed15-a9de-4496-bb25-3064208777f0","Type":"ContainerStarted","Data":"d63d5d80903dbf65ec8492140e491fa139ee6496d9c9b1304b33d70079d4634c"} Jul 23 11:31:14.728721 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:14.728668 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-25z6p" podStartSLOduration=2.00056899 podStartE2EDuration="4.728648841s" totalImagesPullingTime="2.728079851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:14.728102363 +0000 UTC m=+266.745989002" watchObservedRunningTime="2026-07-23 11:31:14.728648841 +0000 UTC m=+266.746535475" Jul 23 11:31:18.438740 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.438680 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b"] Jul 23 11:31:18.443216 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.443184 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.445933 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.445698 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 11:31:18.445933 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.445752 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 11:31:18.445933 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.445807 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 11:31:18.446937 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.446914 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 11:31:18.447154 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.447134 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4vjtg\"" Jul 23 11:31:18.462045 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.462012 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b"] Jul 23 11:31:18.549968 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.549927 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.550224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.549975 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.550224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.550004 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.550224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.550054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.550224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.550087 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pj4\" (UniqueName: \"kubernetes.io/projected/5fbaaa96-64db-4d91-9f11-aa3d25f86695-kube-api-access-97pj4\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.550224 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.550199 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbaaa96-64db-4d91-9f11-aa3d25f86695-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.550398 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.550318 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.651550 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.651514 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.651748 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.651598 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.651748 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.651635 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.651748 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.651660 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.651748 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.651689 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.651748 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.651712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pj4\" (UniqueName: \"kubernetes.io/projected/5fbaaa96-64db-4d91-9f11-aa3d25f86695-kube-api-access-97pj4\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.651990 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.651792 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbaaa96-64db-4d91-9f11-aa3d25f86695-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.652388 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.652300 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.654241 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.654212 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbaaa96-64db-4d91-9f11-aa3d25f86695-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.654351 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.654317 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.654427 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.654408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.654463 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.654451 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fbaaa96-64db-4d91-9f11-aa3d25f86695-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.683840 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.683802 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pj4\" (UniqueName: \"kubernetes.io/projected/5fbaaa96-64db-4d91-9f11-aa3d25f86695-kube-api-access-97pj4\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.685009 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.684982 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbaaa96-64db-4d91-9f11-aa3d25f86695-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-74x8b\" (UID: \"5fbaaa96-64db-4d91-9f11-aa3d25f86695\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.754711 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.754622 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:18.906863 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:18.906837 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b"] Jul 23 11:31:18.909417 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:31:18.909384 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbaaa96_64db_4d91_9f11_aa3d25f86695.slice/crio-d48fd36f6d6d2b9aa33e4cd6505a18958a6e79bf89189426d8815075ebfa3358 WatchSource:0}: Error finding container d48fd36f6d6d2b9aa33e4cd6505a18958a6e79bf89189426d8815075ebfa3358: Status 404 returned error can't find the container with id d48fd36f6d6d2b9aa33e4cd6505a18958a6e79bf89189426d8815075ebfa3358 Jul 23 11:31:19.169796 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.169762 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb"] Jul 23 11:31:19.173992 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.173969 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.177317 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.177284 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 11:31:19.177471 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.177284 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 11:31:19.178058 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.178040 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 11:31:19.178317 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.178302 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-drzrj\"" Jul 23 11:31:19.178373 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.178304 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:31:19.178846 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.178832 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 11:31:19.258121 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.258076 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5tp9\" (UniqueName: \"kubernetes.io/projected/f836c371-db0e-4210-8874-0b1d6ba04522-kube-api-access-n5tp9\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.258366 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.258136 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f836c371-db0e-4210-8874-0b1d6ba04522-manager-config\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.258366 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.258268 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f836c371-db0e-4210-8874-0b1d6ba04522-cert\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.258366 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.258304 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f836c371-db0e-4210-8874-0b1d6ba04522-metrics-cert\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.275049 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.275012 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb"] Jul 23 11:31:19.359444 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.359405 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f836c371-db0e-4210-8874-0b1d6ba04522-cert\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.359444 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.359446 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f836c371-db0e-4210-8874-0b1d6ba04522-metrics-cert\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.359710 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.359638 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5tp9\" (UniqueName: \"kubernetes.io/projected/f836c371-db0e-4210-8874-0b1d6ba04522-kube-api-access-n5tp9\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.359710 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.359694 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f836c371-db0e-4210-8874-0b1d6ba04522-manager-config\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.360411 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.360339 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f836c371-db0e-4210-8874-0b1d6ba04522-manager-config\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.362288 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.362265 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f836c371-db0e-4210-8874-0b1d6ba04522-metrics-cert\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.362288 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.362286 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f836c371-db0e-4210-8874-0b1d6ba04522-cert\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.370437 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.370410 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5tp9\" (UniqueName: \"kubernetes.io/projected/f836c371-db0e-4210-8874-0b1d6ba04522-kube-api-access-n5tp9\") pod \"lws-controller-manager-7c45f5478c-jfttb\" (UID: \"f836c371-db0e-4210-8874-0b1d6ba04522\") " pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.484310 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.484212 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:19.653124 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.653092 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb"] Jul 23 11:31:19.656369 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:31:19.656330 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf836c371_db0e_4210_8874_0b1d6ba04522.slice/crio-91317d06e3db78694583682dd392669db6225823c4c65eeb41d81018d9d73bcd WatchSource:0}: Error finding container 91317d06e3db78694583682dd392669db6225823c4c65eeb41d81018d9d73bcd: Status 404 returned error can't find the container with id 91317d06e3db78694583682dd392669db6225823c4c65eeb41d81018d9d73bcd Jul 23 11:31:19.735240 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.735130 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" event={"ID":"f836c371-db0e-4210-8874-0b1d6ba04522","Type":"ContainerStarted","Data":"91317d06e3db78694583682dd392669db6225823c4c65eeb41d81018d9d73bcd"} Jul 23 11:31:19.736705 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:19.736669 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" event={"ID":"5fbaaa96-64db-4d91-9f11-aa3d25f86695","Type":"ContainerStarted","Data":"d48fd36f6d6d2b9aa33e4cd6505a18958a6e79bf89189426d8815075ebfa3358"} Jul 23 11:31:22.539046 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:22.539003 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 23 11:31:22.539380 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:22.539081 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 23 11:31:22.754551 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:22.754512 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" event={"ID":"5fbaaa96-64db-4d91-9f11-aa3d25f86695","Type":"ContainerStarted","Data":"cb1f208bc506d9c5b77eb4434a091918f8896ddb88343e7c45411f6a36774b73"} Jul 23 11:31:22.754749 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:22.754604 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:22.756019 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:22.755985 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" event={"ID":"f836c371-db0e-4210-8874-0b1d6ba04522","Type":"ContainerStarted","Data":"c00d3e294d1eec9194eacbb8c17f8178a9a5feadc3f91499ce44987d07cf4045"} Jul 23 11:31:22.756135 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:22.756114 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:22.836010 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:22.835951 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" podStartSLOduration=0.900706389 podStartE2EDuration="3.835929545s" totalImagesPullingTime="2.935223156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:22.835778739 +0000 UTC m=+274.853665378" watchObservedRunningTime="2026-07-23 11:31:22.835929545 +0000 UTC m=+274.853816180" Jul 23 11:31:22.837129 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:22.837092 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" podStartSLOduration=1.209794178 podStartE2EDuration="4.837077467s" totalImagesPullingTime="3.627283289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:22.80000026 +0000 UTC m=+274.817886895" watchObservedRunningTime="2026-07-23 11:31:22.837077467 +0000 UTC m=+274.854964102" Jul 23 11:31:23.762344 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:23.762315 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-74x8b" Jul 23 11:31:33.763056 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:33.763021 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c45f5478c-jfttb" Jul 23 11:31:48.350324 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:48.350292 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:31:48.352867 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:48.352843 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:31:48.354703 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:31:48.354682 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 11:35:47.318075 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.318038 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh"] Jul 23 11:35:47.321336 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.321315 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.323870 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.323845 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 11:35:47.324796 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.324771 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 23 11:35:47.324917 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.324814 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 11:35:47.324917 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.324895 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q2mvb\"" Jul 23 11:35:47.333155 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.333122 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh"] Jul 23 11:35:47.339994 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.339958 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.339994 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.340000 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.340283 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.340027 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.340283 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.340108 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q6s9\" (UniqueName: \"kubernetes.io/projected/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-kube-api-access-8q6s9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.340283 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.340264 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.340450 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.340309 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441369 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441369 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441308 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441369 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441357 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441369 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441714 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441714 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6s9\" (UniqueName: \"kubernetes.io/projected/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-kube-api-access-8q6s9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441826 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441826 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441798 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.441920 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.441843 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.444143 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.444117 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.444393 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.444376 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.450515 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.450487 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6s9\" (UniqueName: \"kubernetes.io/projected/33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b-kube-api-access-8q6s9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztbrh\" (UID: \"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.635096 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.635049 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:47.776048 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.776011 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh"] Jul 23 11:35:47.779632 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:35:47.779600 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c8cd5b_7755_4dfd_a4b3_53d1f04ba55b.slice/crio-716c79e0c288ed3a0aea745b8cf28c67772c2a932fc053b15d7c0050c56ef24a WatchSource:0}: Error finding container 716c79e0c288ed3a0aea745b8cf28c67772c2a932fc053b15d7c0050c56ef24a: Status 404 returned error can't find the container with id 716c79e0c288ed3a0aea745b8cf28c67772c2a932fc053b15d7c0050c56ef24a Jul 23 11:35:47.780995 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.780976 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:35:47.902712 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:47.902674 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" event={"ID":"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b","Type":"ContainerStarted","Data":"716c79e0c288ed3a0aea745b8cf28c67772c2a932fc053b15d7c0050c56ef24a"} Jul 23 11:35:48.911670 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:48.911622 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" event={"ID":"33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b","Type":"ContainerStarted","Data":"8944cf0f8b1249b28d098ae4c82777f13fd4372f85b68bba7d9cad9a494516dc"} Jul 23 11:35:48.930924 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:48.930824 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" podStartSLOduration=0.886808891 podStartE2EDuration="1.930795165s" totalImagesPullingTime="1.043986274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:48.93072606 +0000 UTC m=+540.948612693" watchObservedRunningTime="2026-07-23 11:35:48.930795165 +0000 UTC m=+540.948681801" Jul 23 11:35:57.635616 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:57.635547 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:57.636104 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:57.635679 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:57.650110 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:57.650070 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:57.961709 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:57.961618 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztbrh" Jul 23 11:35:59.628821 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.628789 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62"] Jul 23 11:35:59.632375 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.632346 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.634869 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.634844 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 11:35:59.644431 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.644395 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62"] Jul 23 11:35:59.760933 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.760894 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.761127 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.760955 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fc63e6-081d-4596-aaed-5fad30417197-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.761127 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.761004 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.761127 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.761038 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.761127 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.761076 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978vs\" (UniqueName: \"kubernetes.io/projected/08fc63e6-081d-4596-aaed-5fad30417197-kube-api-access-978vs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.761127 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.761119 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.862452 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.862412 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.862662 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.862459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fc63e6-081d-4596-aaed-5fad30417197-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.862662 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.862531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.862662 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.862559 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.862662 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.862624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978vs\" (UniqueName: \"kubernetes.io/projected/08fc63e6-081d-4596-aaed-5fad30417197-kube-api-access-978vs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.862662 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.862652 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.862978 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.862957 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.862978 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.862973 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.863069 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.863010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.865106 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.865079 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fc63e6-081d-4596-aaed-5fad30417197-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.865299 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.865281 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fc63e6-081d-4596-aaed-5fad30417197-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.871020 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.870986 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-978vs\" (UniqueName: \"kubernetes.io/projected/08fc63e6-081d-4596-aaed-5fad30417197-kube-api-access-978vs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62\" (UID: \"08fc63e6-081d-4596-aaed-5fad30417197\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:35:59.947083 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:35:59.946985 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:36:00.086126 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:00.086086 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62"] Jul 23 11:36:00.090313 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:36:00.090283 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fc63e6_081d_4596_aaed_5fad30417197.slice/crio-3090388fd92b5702c2ec41e40f63ee923896ec9be656ab2d495e1650d9509118 WatchSource:0}: Error finding container 3090388fd92b5702c2ec41e40f63ee923896ec9be656ab2d495e1650d9509118: Status 404 returned error can't find the container with id 3090388fd92b5702c2ec41e40f63ee923896ec9be656ab2d495e1650d9509118 Jul 23 11:36:00.965026 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:00.964989 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" event={"ID":"08fc63e6-081d-4596-aaed-5fad30417197","Type":"ContainerStarted","Data":"ea2f895aeb2463552cf8080d0ff92ce299f34acc770ae069e5235cba8e97107d"} Jul 23 11:36:00.965414 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:00.965072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" event={"ID":"08fc63e6-081d-4596-aaed-5fad30417197","Type":"ContainerStarted","Data":"3090388fd92b5702c2ec41e40f63ee923896ec9be656ab2d495e1650d9509118"} Jul 23 11:36:00.984920 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:00.984867 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" podStartSLOduration=1.7886814580000001 podStartE2EDuration="1.984850628s" totalImagesPullingTime="196.16917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:00.983677169 +0000 UTC m=+553.001563804" watchObservedRunningTime="2026-07-23 11:36:00.984850628 +0000 UTC m=+553.002737283" Jul 23 11:36:09.947431 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:09.947385 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:36:09.947890 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:09.947458 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:36:09.959972 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:09.959945 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:36:10.014285 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:10.014241 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62" Jul 23 11:36:48.378311 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:48.378283 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:36:48.379267 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:36:48.379247 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:41:48.403396 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:41:48.403308 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:41:48.406377 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:41:48.406350 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:46:48.427625 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:46:48.427591 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:46:48.430885 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:46:48.430867 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:48:10.824831 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.824798 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t"] Jul 23 11:48:10.828006 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.827990 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:10.832057 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.832036 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-694b89c1-kserve-self-signed-certs\"" Jul 23 11:48:10.839969 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.839947 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t"] Jul 23 11:48:10.974690 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.974660 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-dshm\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:10.974690 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.974694 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-tmp-dir\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:10.974894 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.974717 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4ff\" (UniqueName: \"kubernetes.io/projected/b562f4a5-4606-4503-9661-c535a23970c2-kube-api-access-kt4ff\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:10.974894 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.974812 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b562f4a5-4606-4503-9661-c535a23970c2-tls-certs\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:10.974965 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.974895 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-home\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:10.974965 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:10.974919 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-model-cache\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076028 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.075942 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4ff\" (UniqueName: \"kubernetes.io/projected/b562f4a5-4606-4503-9661-c535a23970c2-kube-api-access-kt4ff\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076028 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.075994 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b562f4a5-4606-4503-9661-c535a23970c2-tls-certs\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076235 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.076055 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-home\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076235 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.076086 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-model-cache\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076235 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.076142 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-dshm\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076235 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.076163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-tmp-dir\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076672 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.076640 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-model-cache\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076811 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.076653 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-tmp-dir\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.076811 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.076690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-home\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.078384 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.078362 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-dshm\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.078775 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.078756 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b562f4a5-4606-4503-9661-c535a23970c2-tls-certs\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.083316 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.083297 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4ff\" (UniqueName: \"kubernetes.io/projected/b562f4a5-4606-4503-9661-c535a23970c2-kube-api-access-kt4ff\") pod \"e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.137528 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.137489 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:11.267188 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:11.267166 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t"] Jul 23 11:48:11.269688 ip-10-0-129-204 kubenswrapper[2556]: W0723 11:48:11.269665 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb562f4a5_4606_4503_9661_c535a23970c2.slice/crio-dc13b15872b6b576420468277c966ca1c193abfdd60519cec74d6c3f2b1391dc WatchSource:0}: Error finding container dc13b15872b6b576420468277c966ca1c193abfdd60519cec74d6c3f2b1391dc: Status 404 returned error can't find the container with id dc13b15872b6b576420468277c966ca1c193abfdd60519cec74d6c3f2b1391dc Jul 23 11:48:12.097051 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:12.097020 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" event={"ID":"b562f4a5-4606-4503-9661-c535a23970c2","Type":"ContainerStarted","Data":"f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489"} Jul 23 11:48:12.097488 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:12.097065 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" event={"ID":"b562f4a5-4606-4503-9661-c535a23970c2","Type":"ContainerStarted","Data":"dc13b15872b6b576420468277c966ca1c193abfdd60519cec74d6c3f2b1391dc"} Jul 23 11:48:12.117863 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:12.117817 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" podStartSLOduration=2.117802932 podStartE2EDuration="2.117802932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:48:12.116759573 +0000 UTC m=+1284.134646208" watchObservedRunningTime="2026-07-23 11:48:12.117802932 +0000 UTC m=+1284.135689597" Jul 23 11:48:21.138117 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:21.138082 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:21.138500 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:21.138160 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:21.150145 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:21.150116 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:22.151135 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:22.151108 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:37.636816 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:37.636782 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t"] Jul 23 11:48:52.659525 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:52.659484 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" podUID="b562f4a5-4606-4503-9661-c535a23970c2" containerName="main" containerID="cri-o://f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489" gracePeriod=45 Jul 23 11:48:52.897630 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:52.897606 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:53.015312 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015238 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-home\") pod \"b562f4a5-4606-4503-9661-c535a23970c2\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " Jul 23 11:48:53.015312 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015283 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-model-cache\") pod \"b562f4a5-4606-4503-9661-c535a23970c2\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " Jul 23 11:48:53.015312 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015303 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b562f4a5-4606-4503-9661-c535a23970c2-tls-certs\") pod \"b562f4a5-4606-4503-9661-c535a23970c2\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " Jul 23 11:48:53.015593 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015334 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-dshm\") pod \"b562f4a5-4606-4503-9661-c535a23970c2\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " Jul 23 11:48:53.015593 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015360 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-tmp-dir\") pod \"b562f4a5-4606-4503-9661-c535a23970c2\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " Jul 23 11:48:53.015593 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015415 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt4ff\" (UniqueName: \"kubernetes.io/projected/b562f4a5-4606-4503-9661-c535a23970c2-kube-api-access-kt4ff\") pod \"b562f4a5-4606-4503-9661-c535a23970c2\" (UID: \"b562f4a5-4606-4503-9661-c535a23970c2\") " Jul 23 11:48:53.015593 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015529 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "b562f4a5-4606-4503-9661-c535a23970c2" (UID: "b562f4a5-4606-4503-9661-c535a23970c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:48:53.015845 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015563 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-home" (OuterVolumeSpecName: "home") pod "b562f4a5-4606-4503-9661-c535a23970c2" (UID: "b562f4a5-4606-4503-9661-c535a23970c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:48:53.015845 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015661 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:48:53.015845 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.015669 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b562f4a5-4606-4503-9661-c535a23970c2" (UID: "b562f4a5-4606-4503-9661-c535a23970c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:48:53.017720 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.017696 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b562f4a5-4606-4503-9661-c535a23970c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b562f4a5-4606-4503-9661-c535a23970c2" (UID: "b562f4a5-4606-4503-9661-c535a23970c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:48:53.017801 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.017772 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b562f4a5-4606-4503-9661-c535a23970c2-kube-api-access-kt4ff" (OuterVolumeSpecName: "kube-api-access-kt4ff") pod "b562f4a5-4606-4503-9661-c535a23970c2" (UID: "b562f4a5-4606-4503-9661-c535a23970c2"). InnerVolumeSpecName "kube-api-access-kt4ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:48:53.017850 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.017800 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-dshm" (OuterVolumeSpecName: "dshm") pod "b562f4a5-4606-4503-9661-c535a23970c2" (UID: "b562f4a5-4606-4503-9661-c535a23970c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:48:53.116372 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.116342 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4ff\" (UniqueName: \"kubernetes.io/projected/b562f4a5-4606-4503-9661-c535a23970c2-kube-api-access-kt4ff\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:48:53.116372 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.116368 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:48:53.116521 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.116382 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b562f4a5-4606-4503-9661-c535a23970c2-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:48:53.116521 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.116394 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:48:53.116521 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.116428 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b562f4a5-4606-4503-9661-c535a23970c2-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 11:48:53.270050 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.269965 2556 generic.go:356] "Generic (PLEG): container finished" podID="b562f4a5-4606-4503-9661-c535a23970c2" containerID="f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489" exitCode=0 Jul 23 11:48:53.270050 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.270014 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" Jul 23 11:48:53.270225 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.270050 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" event={"ID":"b562f4a5-4606-4503-9661-c535a23970c2","Type":"ContainerDied","Data":"f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489"} Jul 23 11:48:53.270225 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.270089 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t" event={"ID":"b562f4a5-4606-4503-9661-c535a23970c2","Type":"ContainerDied","Data":"dc13b15872b6b576420468277c966ca1c193abfdd60519cec74d6c3f2b1391dc"} Jul 23 11:48:53.270225 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.270105 2556 scope.go:117] "RemoveContainer" containerID="f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489" Jul 23 11:48:53.278980 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.278963 2556 scope.go:117] "RemoveContainer" containerID="f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489" Jul 23 11:48:53.279241 ip-10-0-129-204 kubenswrapper[2556]: E0723 11:48:53.279224 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489\": container with ID starting with f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489 not found: ID does not exist" containerID="f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489" Jul 23 11:48:53.279296 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.279251 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489"} err="failed to get container status \"f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489\": rpc error: code = NotFound desc = could not find container \"f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489\": container with ID starting with f429b49521b4d14079fd21054369a00d19668333fd88c96f9c3b9bbd3233c489 not found: ID does not exist" Jul 23 11:48:53.288974 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.288953 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t"] Jul 23 11:48:53.292843 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:53.292823 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-694b89c1-kserve-655b8cdf59-hnl2t"] Jul 23 11:48:54.501152 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:48:54.501119 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b562f4a5-4606-4503-9661-c535a23970c2" path="/var/lib/kubelet/pods/b562f4a5-4606-4503-9661-c535a23970c2/volumes" Jul 23 11:51:48.459097 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:51:48.459070 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:51:48.462257 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:51:48.462237 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:56:48.483991 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:56:48.483967 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 11:56:48.487288 ip-10-0-129-204 kubenswrapper[2556]: I0723 11:56:48.487269 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 12:01:28.362166 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.362129 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q"] Jul 23 12:01:28.362598 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.362534 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b562f4a5-4606-4503-9661-c535a23970c2" containerName="main" Jul 23 12:01:28.362598 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.362546 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b562f4a5-4606-4503-9661-c535a23970c2" containerName="main" Jul 23 12:01:28.362675 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.362615 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b562f4a5-4606-4503-9661-c535a23970c2" containerName="main" Jul 23 12:01:28.365347 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.365330 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.368790 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.368763 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f8adc6\"/\"test-model-f8adc6-kserve-self-signed-certs\"" Jul 23 12:01:28.368790 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.368779 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f8adc6\"/\"openshift-service-ca.crt\"" Jul 23 12:01:28.368997 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.368763 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f8adc6\"/\"kube-root-ca.crt\"" Jul 23 12:01:28.368997 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.368819 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f8adc6\"/\"default-dockercfg-hwxlm\"" Jul 23 12:01:28.378830 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.378803 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q"] Jul 23 12:01:28.447956 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.447934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-home\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.448081 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.447964 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-dshm\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.448081 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.447981 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbc742-f77c-4609-bd04-312bb65ff33a-tls-certs\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.448081 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.448000 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-tmp-dir\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.448081 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.448020 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-model-cache\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.448215 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.448163 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdstt\" (UniqueName: \"kubernetes.io/projected/a2dbc742-f77c-4609-bd04-312bb65ff33a-kube-api-access-sdstt\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.549499 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.549473 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdstt\" (UniqueName: \"kubernetes.io/projected/a2dbc742-f77c-4609-bd04-312bb65ff33a-kube-api-access-sdstt\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.549624 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.549513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-home\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.549624 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.549540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-dshm\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.549624 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.549563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbc742-f77c-4609-bd04-312bb65ff33a-tls-certs\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.549624 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.549617 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-tmp-dir\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.549827 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.549643 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-model-cache\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.549921 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.549900 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-home\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.550039 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.550021 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-model-cache\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.550039 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.550033 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-tmp-dir\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.551995 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.551975 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-dshm\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.552292 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.552260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbc742-f77c-4609-bd04-312bb65ff33a-tls-certs\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.557474 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.557454 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdstt\" (UniqueName: \"kubernetes.io/projected/a2dbc742-f77c-4609-bd04-312bb65ff33a-kube-api-access-sdstt\") pod \"test-model-f8adc6-kserve-7669f6488f-2p82q\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.675045 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.674986 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:28.803714 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:28.803686 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q"] Jul 23 12:01:28.806328 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:01:28.806299 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2dbc742_f77c_4609_bd04_312bb65ff33a.slice/crio-db7468b3eec753d77e4c6b822d9734cfc482b00dc0f40bc186b9aa15c350fb6b WatchSource:0}: Error finding container db7468b3eec753d77e4c6b822d9734cfc482b00dc0f40bc186b9aa15c350fb6b: Status 404 returned error can't find the container with id db7468b3eec753d77e4c6b822d9734cfc482b00dc0f40bc186b9aa15c350fb6b Jul 23 12:01:29.507733 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:29.507698 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" event={"ID":"a2dbc742-f77c-4609-bd04-312bb65ff33a","Type":"ContainerStarted","Data":"bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9"} Jul 23 12:01:29.507733 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:29.507741 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" event={"ID":"a2dbc742-f77c-4609-bd04-312bb65ff33a","Type":"ContainerStarted","Data":"db7468b3eec753d77e4c6b822d9734cfc482b00dc0f40bc186b9aa15c350fb6b"} Jul 23 12:01:29.523665 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:29.523619 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" podStartSLOduration=1.5236037279999999 podStartE2EDuration="1.523603728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:01:29.522722193 +0000 UTC m=+2081.540608854" watchObservedRunningTime="2026-07-23 12:01:29.523603728 +0000 UTC m=+2081.541490364" Jul 23 12:01:38.675364 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:38.675330 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:38.675843 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:38.675405 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:38.687772 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:38.687746 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:39.561348 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:39.561318 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:01:48.510848 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:48.510819 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 12:01:48.513183 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:01:48.513166 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 12:02:28.546257 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:28.545536 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q"] Jul 23 12:02:43.570551 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.570458 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" podUID="a2dbc742-f77c-4609-bd04-312bb65ff33a" containerName="main" containerID="cri-o://bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9" gracePeriod=45 Jul 23 12:02:43.810940 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.810907 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:02:43.833801 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.833770 2556 generic.go:356] "Generic (PLEG): container finished" podID="a2dbc742-f77c-4609-bd04-312bb65ff33a" containerID="bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9" exitCode=0 Jul 23 12:02:43.833920 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.833812 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" Jul 23 12:02:43.833920 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.833847 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" event={"ID":"a2dbc742-f77c-4609-bd04-312bb65ff33a","Type":"ContainerDied","Data":"bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9"} Jul 23 12:02:43.833920 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.833893 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q" event={"ID":"a2dbc742-f77c-4609-bd04-312bb65ff33a","Type":"ContainerDied","Data":"db7468b3eec753d77e4c6b822d9734cfc482b00dc0f40bc186b9aa15c350fb6b"} Jul 23 12:02:43.833920 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.833916 2556 scope.go:117] "RemoveContainer" containerID="bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9" Jul 23 12:02:43.841880 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.841855 2556 scope.go:117] "RemoveContainer" containerID="bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9" Jul 23 12:02:43.842147 ip-10-0-129-204 kubenswrapper[2556]: E0723 12:02:43.842126 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9\": container with ID starting with bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9 not found: ID does not exist" containerID="bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9" Jul 23 12:02:43.842211 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.842156 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9"} err="failed to get container status \"bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9\": rpc error: code = NotFound desc = could not find container \"bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9\": container with ID starting with bce8114ba8d54a541377b97db7fd8b8303084003bd4fb1e854f2597c11f9c0f9 not found: ID does not exist" Jul 23 12:02:43.923115 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923093 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-model-cache\") pod \"a2dbc742-f77c-4609-bd04-312bb65ff33a\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " Jul 23 12:02:43.923198 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923121 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-home\") pod \"a2dbc742-f77c-4609-bd04-312bb65ff33a\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " Jul 23 12:02:43.923198 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923172 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-tmp-dir\") pod \"a2dbc742-f77c-4609-bd04-312bb65ff33a\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " Jul 23 12:02:43.923278 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923203 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbc742-f77c-4609-bd04-312bb65ff33a-tls-certs\") pod \"a2dbc742-f77c-4609-bd04-312bb65ff33a\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " Jul 23 12:02:43.923278 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923253 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdstt\" (UniqueName: \"kubernetes.io/projected/a2dbc742-f77c-4609-bd04-312bb65ff33a-kube-api-access-sdstt\") pod \"a2dbc742-f77c-4609-bd04-312bb65ff33a\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " Jul 23 12:02:43.923378 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923278 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-dshm\") pod \"a2dbc742-f77c-4609-bd04-312bb65ff33a\" (UID: \"a2dbc742-f77c-4609-bd04-312bb65ff33a\") " Jul 23 12:02:43.923378 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923344 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2dbc742-f77c-4609-bd04-312bb65ff33a" (UID: "a2dbc742-f77c-4609-bd04-312bb65ff33a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:43.923483 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923391 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-home" (OuterVolumeSpecName: "home") pod "a2dbc742-f77c-4609-bd04-312bb65ff33a" (UID: "a2dbc742-f77c-4609-bd04-312bb65ff33a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:43.923483 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923471 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2dbc742-f77c-4609-bd04-312bb65ff33a" (UID: "a2dbc742-f77c-4609-bd04-312bb65ff33a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:43.923616 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923600 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:02:43.923659 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923620 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:02:43.923659 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.923632 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:02:43.925458 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.925436 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-dshm" (OuterVolumeSpecName: "dshm") pod "a2dbc742-f77c-4609-bd04-312bb65ff33a" (UID: "a2dbc742-f77c-4609-bd04-312bb65ff33a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:43.925529 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.925448 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dbc742-f77c-4609-bd04-312bb65ff33a-kube-api-access-sdstt" (OuterVolumeSpecName: "kube-api-access-sdstt") pod "a2dbc742-f77c-4609-bd04-312bb65ff33a" (UID: "a2dbc742-f77c-4609-bd04-312bb65ff33a"). InnerVolumeSpecName "kube-api-access-sdstt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:02:43.925529 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:43.925486 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2dbc742-f77c-4609-bd04-312bb65ff33a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2dbc742-f77c-4609-bd04-312bb65ff33a" (UID: "a2dbc742-f77c-4609-bd04-312bb65ff33a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:02:44.024522 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:44.024501 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbc742-f77c-4609-bd04-312bb65ff33a-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:02:44.024522 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:44.024521 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdstt\" (UniqueName: \"kubernetes.io/projected/a2dbc742-f77c-4609-bd04-312bb65ff33a-kube-api-access-sdstt\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:02:44.024663 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:44.024530 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbc742-f77c-4609-bd04-312bb65ff33a-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:02:44.155282 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:44.155261 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q"] Jul 23 12:02:44.159108 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:44.159088 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f8adc6/test-model-f8adc6-kserve-7669f6488f-2p82q"] Jul 23 12:02:44.499051 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:02:44.498993 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2dbc742-f77c-4609-bd04-312bb65ff33a" path="/var/lib/kubelet/pods/a2dbc742-f77c-4609-bd04-312bb65ff33a/volumes" Jul 23 12:03:38.085858 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.085820 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm"] Jul 23 12:03:38.086295 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.086126 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2dbc742-f77c-4609-bd04-312bb65ff33a" containerName="main" Jul 23 12:03:38.086295 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.086137 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2dbc742-f77c-4609-bd04-312bb65ff33a" containerName="main" Jul 23 12:03:38.086295 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.086207 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2dbc742-f77c-4609-bd04-312bb65ff33a" containerName="main" Jul 23 12:03:38.089024 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.089005 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" Jul 23 12:03:38.091144 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.091115 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 12:03:38.091346 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.091332 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:03:38.097158 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.097121 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm"] Jul 23 12:03:38.137506 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.137482 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43f746-z9vcm\" (UID: \"6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" Jul 23 12:03:38.238691 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.238658 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43f746-z9vcm\" (UID: \"6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" Jul 23 12:03:38.239315 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.239290 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43f746-z9vcm\" (UID: \"6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" Jul 23 12:03:38.400008 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.399949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" Jul 23 12:03:38.530624 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.530592 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm"] Jul 23 12:03:38.534538 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:03:38.534502 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d79712e_fa38_4ad3_aa3b_70eb3a0f9c4b.slice/crio-2663c4618459560561cd3b2bbb37678425a08c35b54c92885c8d84476ae0c90c WatchSource:0}: Error finding container 2663c4618459560561cd3b2bbb37678425a08c35b54c92885c8d84476ae0c90c: Status 404 returned error can't find the container with id 2663c4618459560561cd3b2bbb37678425a08c35b54c92885c8d84476ae0c90c Jul 23 12:03:38.536315 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:38.536299 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 12:03:39.062845 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:39.062814 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" event={"ID":"6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b","Type":"ContainerStarted","Data":"2663c4618459560561cd3b2bbb37678425a08c35b54c92885c8d84476ae0c90c"} Jul 23 12:03:42.079345 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:42.079310 2556 generic.go:356] "Generic (PLEG): container finished" podID="6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b" containerID="eaf8e0d56409504e0124375dcd979bdac1d1952b29007c764ee1af70413d9476" exitCode=0 Jul 23 12:03:42.079768 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:42.079374 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" event={"ID":"6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b","Type":"ContainerDied","Data":"eaf8e0d56409504e0124375dcd979bdac1d1952b29007c764ee1af70413d9476"} Jul 23 12:03:43.211270 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:43.211244 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" Jul 23 12:03:43.276734 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:43.276709 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b-maas-api-service-ca\") pod \"6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b\" (UID: \"6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b\") " Jul 23 12:03:43.277034 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:43.277007 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b" (UID: "6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:03:43.378259 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:43.378201 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b-maas-api-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:03:44.089102 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:44.089071 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" event={"ID":"6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b","Type":"ContainerDied","Data":"2663c4618459560561cd3b2bbb37678425a08c35b54c92885c8d84476ae0c90c"} Jul 23 12:03:44.089102 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:44.089096 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm" Jul 23 12:03:44.089304 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:44.089120 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2663c4618459560561cd3b2bbb37678425a08c35b54c92885c8d84476ae0c90c" Jul 23 12:03:58.180070 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:58.180032 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm"] Jul 23 12:03:58.183914 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:58.183884 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43f746-z9vcm"] Jul 23 12:03:58.499705 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:03:58.499633 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b" path="/var/lib/kubelet/pods/6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b/volumes" Jul 23 12:04:05.013307 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.013271 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88"] Jul 23 12:04:05.013683 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.013607 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b" containerName="revoke-keys" Jul 23 12:04:05.013683 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.013618 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b" containerName="revoke-keys" Jul 23 12:04:05.013683 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.013684 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d79712e-fa38-4ad3-aa3b-70eb3a0f9c4b" containerName="revoke-keys" Jul 23 12:04:05.020517 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.020496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" Jul 23 12:04:05.022827 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.022804 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:04:05.022936 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.022847 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 12:04:05.027238 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.027217 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88"] Jul 23 12:04:05.146960 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.146935 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a61916-5b57-41e7-9982-aeb6aed413dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1f99f-gct88\" (UID: \"f7a61916-5b57-41e7-9982-aeb6aed413dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" Jul 23 12:04:05.247960 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.247929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a61916-5b57-41e7-9982-aeb6aed413dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1f99f-gct88\" (UID: \"f7a61916-5b57-41e7-9982-aeb6aed413dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" Jul 23 12:04:05.248554 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.248529 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a61916-5b57-41e7-9982-aeb6aed413dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1f99f-gct88\" (UID: \"f7a61916-5b57-41e7-9982-aeb6aed413dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" Jul 23 12:04:05.331944 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.331923 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" Jul 23 12:04:05.462107 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:05.462080 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88"] Jul 23 12:04:05.464747 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:04:05.464712 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a61916_5b57_41e7_9982_aeb6aed413dd.slice/crio-f80a87d3ae17d2fe4fbc4d93367bbfc0ba2742cddd6217f5e525dc40ed5fecb3 WatchSource:0}: Error finding container f80a87d3ae17d2fe4fbc4d93367bbfc0ba2742cddd6217f5e525dc40ed5fecb3: Status 404 returned error can't find the container with id f80a87d3ae17d2fe4fbc4d93367bbfc0ba2742cddd6217f5e525dc40ed5fecb3 Jul 23 12:04:06.190066 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:06.190041 2556 generic.go:356] "Generic (PLEG): container finished" podID="f7a61916-5b57-41e7-9982-aeb6aed413dd" containerID="07072b9557aaa25fd677f54589cb86354aef380b33843a92427905762687bee8" exitCode=0 Jul 23 12:04:06.190364 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:06.190132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" event={"ID":"f7a61916-5b57-41e7-9982-aeb6aed413dd","Type":"ContainerDied","Data":"07072b9557aaa25fd677f54589cb86354aef380b33843a92427905762687bee8"} Jul 23 12:04:06.190364 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:06.190192 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" event={"ID":"f7a61916-5b57-41e7-9982-aeb6aed413dd","Type":"ContainerStarted","Data":"f80a87d3ae17d2fe4fbc4d93367bbfc0ba2742cddd6217f5e525dc40ed5fecb3"} Jul 23 12:04:07.318838 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:07.318809 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" Jul 23 12:04:07.366894 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:07.366867 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a61916-5b57-41e7-9982-aeb6aed413dd-maas-api-service-ca\") pod \"f7a61916-5b57-41e7-9982-aeb6aed413dd\" (UID: \"f7a61916-5b57-41e7-9982-aeb6aed413dd\") " Jul 23 12:04:07.367229 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:07.367207 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a61916-5b57-41e7-9982-aeb6aed413dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7a61916-5b57-41e7-9982-aeb6aed413dd" (UID: "f7a61916-5b57-41e7-9982-aeb6aed413dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:04:07.468014 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:07.467941 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a61916-5b57-41e7-9982-aeb6aed413dd-maas-api-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:04:08.199621 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:08.199512 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" event={"ID":"f7a61916-5b57-41e7-9982-aeb6aed413dd","Type":"ContainerDied","Data":"f80a87d3ae17d2fe4fbc4d93367bbfc0ba2742cddd6217f5e525dc40ed5fecb3"} Jul 23 12:04:08.199621 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:08.199535 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88" Jul 23 12:04:08.199621 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:08.199594 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f80a87d3ae17d2fe4fbc4d93367bbfc0ba2742cddd6217f5e525dc40ed5fecb3" Jul 23 12:04:25.112430 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:25.112392 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88"] Jul 23 12:04:25.114386 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:25.114361 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1f99f-gct88"] Jul 23 12:04:26.499801 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:04:26.499761 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7a61916-5b57-41e7-9982-aeb6aed413dd" path="/var/lib/kubelet/pods/f7a61916-5b57-41e7-9982-aeb6aed413dd/volumes" Jul 23 12:05:32.051760 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.051721 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft"] Jul 23 12:05:32.052231 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.052042 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a61916-5b57-41e7-9982-aeb6aed413dd" containerName="revoke-keys" Jul 23 12:05:32.052231 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.052052 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a61916-5b57-41e7-9982-aeb6aed413dd" containerName="revoke-keys" Jul 23 12:05:32.052231 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.052111 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a61916-5b57-41e7-9982-aeb6aed413dd" containerName="revoke-keys" Jul 23 12:05:32.054556 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.054539 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.057068 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.057047 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-77f87c\"/\"kube-root-ca.crt\"" Jul 23 12:05:32.057754 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.057732 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-77f87c\"/\"openshift-service-ca.crt\"" Jul 23 12:05:32.057831 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.057811 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-77f87c\"/\"default-dockercfg-xx7rr\"" Jul 23 12:05:32.058029 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.058015 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-77f87c\"/\"auth-test-model-77f87c-kserve-self-signed-certs\"" Jul 23 12:05:32.064168 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.064142 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft"] Jul 23 12:05:32.108368 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.108343 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-tmp-dir\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.108469 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.108379 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-home\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.108469 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.108412 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-model-cache\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.108544 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.108473 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hsc\" (UniqueName: \"kubernetes.io/projected/e028aa09-a255-4bd5-9756-4edaa721ef7f-kube-api-access-v8hsc\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.108544 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.108527 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e028aa09-a255-4bd5-9756-4edaa721ef7f-tls-certs\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.108639 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.108547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-dshm\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.209743 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.209719 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-tmp-dir\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.209852 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.209758 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-home\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.209852 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.209792 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-model-cache\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.209972 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.209919 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hsc\" (UniqueName: \"kubernetes.io/projected/e028aa09-a255-4bd5-9756-4edaa721ef7f-kube-api-access-v8hsc\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.209972 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.209965 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e028aa09-a255-4bd5-9756-4edaa721ef7f-tls-certs\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.210079 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.209995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-dshm\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.210224 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.210201 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-model-cache\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.210314 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.210203 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-tmp-dir\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.210314 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.210252 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-home\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.212414 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.212397 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-dshm\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.212716 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.212701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e028aa09-a255-4bd5-9756-4edaa721ef7f-tls-certs\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.218287 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.218263 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hsc\" (UniqueName: \"kubernetes.io/projected/e028aa09-a255-4bd5-9756-4edaa721ef7f-kube-api-access-v8hsc\") pod \"auth-test-model-77f87c-kserve-7f55dc8db7-q69ft\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.364381 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.364362 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:32.500024 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.499997 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft"] Jul 23 12:05:32.503292 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:05:32.503261 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode028aa09_a255_4bd5_9756_4edaa721ef7f.slice/crio-f1c36b869a474bf2d6dcb19950a2ff24225689a38ec4a9cc35894e96a7930bf7 WatchSource:0}: Error finding container f1c36b869a474bf2d6dcb19950a2ff24225689a38ec4a9cc35894e96a7930bf7: Status 404 returned error can't find the container with id f1c36b869a474bf2d6dcb19950a2ff24225689a38ec4a9cc35894e96a7930bf7 Jul 23 12:05:32.572728 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.572698 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" event={"ID":"e028aa09-a255-4bd5-9756-4edaa721ef7f","Type":"ContainerStarted","Data":"a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e"} Jul 23 12:05:32.572827 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.572744 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" event={"ID":"e028aa09-a255-4bd5-9756-4edaa721ef7f","Type":"ContainerStarted","Data":"f1c36b869a474bf2d6dcb19950a2ff24225689a38ec4a9cc35894e96a7930bf7"} Jul 23 12:05:32.590350 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:32.590291 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" podStartSLOduration=0.590272844 podStartE2EDuration="590.272844ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:05:32.589733099 +0000 UTC m=+2324.607619733" watchObservedRunningTime="2026-07-23 12:05:32.590272844 +0000 UTC m=+2324.608159480" Jul 23 12:05:42.365243 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:42.365150 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:42.365243 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:42.365207 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:42.378139 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:42.378110 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:05:42.629291 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:05:42.629204 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:06:48.541955 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:06:48.541922 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 12:06:48.545028 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:06:48.545010 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 12:07:39.150548 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:39.150458 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft"] Jul 23 12:07:54.170845 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.170774 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" podUID="e028aa09-a255-4bd5-9756-4edaa721ef7f" containerName="main" containerID="cri-o://a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e" gracePeriod=45 Jul 23 12:07:54.425097 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.425046 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:07:54.481687 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.481666 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-home\") pod \"e028aa09-a255-4bd5-9756-4edaa721ef7f\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " Jul 23 12:07:54.481799 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.481715 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8hsc\" (UniqueName: \"kubernetes.io/projected/e028aa09-a255-4bd5-9756-4edaa721ef7f-kube-api-access-v8hsc\") pod \"e028aa09-a255-4bd5-9756-4edaa721ef7f\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " Jul 23 12:07:54.481799 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.481734 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-tmp-dir\") pod \"e028aa09-a255-4bd5-9756-4edaa721ef7f\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " Jul 23 12:07:54.481799 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.481764 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e028aa09-a255-4bd5-9756-4edaa721ef7f-tls-certs\") pod \"e028aa09-a255-4bd5-9756-4edaa721ef7f\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " Jul 23 12:07:54.481913 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.481868 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-model-cache\") pod \"e028aa09-a255-4bd5-9756-4edaa721ef7f\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " Jul 23 12:07:54.481913 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.481893 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-dshm\") pod \"e028aa09-a255-4bd5-9756-4edaa721ef7f\" (UID: \"e028aa09-a255-4bd5-9756-4edaa721ef7f\") " Jul 23 12:07:54.482015 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.481974 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-home" (OuterVolumeSpecName: "home") pod "e028aa09-a255-4bd5-9756-4edaa721ef7f" (UID: "e028aa09-a255-4bd5-9756-4edaa721ef7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:07:54.482015 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.481992 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e028aa09-a255-4bd5-9756-4edaa721ef7f" (UID: "e028aa09-a255-4bd5-9756-4edaa721ef7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:07:54.482158 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.482136 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e028aa09-a255-4bd5-9756-4edaa721ef7f" (UID: "e028aa09-a255-4bd5-9756-4edaa721ef7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:07:54.482211 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.482160 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:07:54.482211 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.482178 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:07:54.483993 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.483970 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-dshm" (OuterVolumeSpecName: "dshm") pod "e028aa09-a255-4bd5-9756-4edaa721ef7f" (UID: "e028aa09-a255-4bd5-9756-4edaa721ef7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:07:54.484069 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.483991 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e028aa09-a255-4bd5-9756-4edaa721ef7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e028aa09-a255-4bd5-9756-4edaa721ef7f" (UID: "e028aa09-a255-4bd5-9756-4edaa721ef7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:07:54.484117 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.484061 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e028aa09-a255-4bd5-9756-4edaa721ef7f-kube-api-access-v8hsc" (OuterVolumeSpecName: "kube-api-access-v8hsc") pod "e028aa09-a255-4bd5-9756-4edaa721ef7f" (UID: "e028aa09-a255-4bd5-9756-4edaa721ef7f"). InnerVolumeSpecName "kube-api-access-v8hsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:07:54.583244 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.583222 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:07:54.583244 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.583242 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e028aa09-a255-4bd5-9756-4edaa721ef7f-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:07:54.583364 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.583251 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8hsc\" (UniqueName: \"kubernetes.io/projected/e028aa09-a255-4bd5-9756-4edaa721ef7f-kube-api-access-v8hsc\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:07:54.583364 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:54.583259 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e028aa09-a255-4bd5-9756-4edaa721ef7f-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:07:55.207975 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.207939 2556 generic.go:356] "Generic (PLEG): container finished" podID="e028aa09-a255-4bd5-9756-4edaa721ef7f" containerID="a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e" exitCode=0 Jul 23 12:07:55.208433 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.207981 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" Jul 23 12:07:55.208433 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.208030 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" event={"ID":"e028aa09-a255-4bd5-9756-4edaa721ef7f","Type":"ContainerDied","Data":"a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e"} Jul 23 12:07:55.208433 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.208068 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft" event={"ID":"e028aa09-a255-4bd5-9756-4edaa721ef7f","Type":"ContainerDied","Data":"f1c36b869a474bf2d6dcb19950a2ff24225689a38ec4a9cc35894e96a7930bf7"} Jul 23 12:07:55.208433 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.208089 2556 scope.go:117] "RemoveContainer" containerID="a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e" Jul 23 12:07:55.216796 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.216778 2556 scope.go:117] "RemoveContainer" containerID="a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e" Jul 23 12:07:55.217074 ip-10-0-129-204 kubenswrapper[2556]: E0723 12:07:55.217055 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e\": container with ID starting with a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e not found: ID does not exist" containerID="a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e" Jul 23 12:07:55.217118 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.217084 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e"} err="failed to get container status \"a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e\": rpc error: code = NotFound desc = could not find container \"a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e\": container with ID starting with a261d4bd5eddd535c82c9148d5457a47ef8546bdb69bdf6803463a4abd757a4e not found: ID does not exist" Jul 23 12:07:55.232691 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.231630 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft"] Jul 23 12:07:55.235908 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:55.235886 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/auth-test-model-77f87c-kserve-7f55dc8db7-q69ft"] Jul 23 12:07:56.049619 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.049565 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58"] Jul 23 12:07:56.049982 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.049964 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e028aa09-a255-4bd5-9756-4edaa721ef7f" containerName="main" Jul 23 12:07:56.050068 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.049984 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e028aa09-a255-4bd5-9756-4edaa721ef7f" containerName="main" Jul 23 12:07:56.050120 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.050089 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e028aa09-a255-4bd5-9756-4edaa721ef7f" containerName="main" Jul 23 12:07:56.054451 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.054428 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.057015 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.056998 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7cc2e5\"/\"openshift-service-ca.crt\"" Jul 23 12:07:56.057664 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.057649 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7cc2e5\"/\"kube-root-ca.crt\"" Jul 23 12:07:56.057740 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.057724 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7cc2e5\"/\"default-dockercfg-6scws\"" Jul 23 12:07:56.057828 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.057811 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7cc2e5\"/\"sub-test-model-7cc2e5-kserve-self-signed-certs\"" Jul 23 12:07:56.064258 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.064237 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58"] Jul 23 12:07:56.096015 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.095985 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-model-cache\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.096125 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.096023 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b878a7-3169-4390-9184-79f9ff5e642f-tls-certs\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.096125 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.096066 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-tmp-dir\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.096235 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.096134 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-home\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.096235 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.096174 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-dshm\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.096339 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.096231 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jq5f\" (UniqueName: \"kubernetes.io/projected/d3b878a7-3169-4390-9184-79f9ff5e642f-kube-api-access-8jq5f\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197086 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jq5f\" (UniqueName: \"kubernetes.io/projected/d3b878a7-3169-4390-9184-79f9ff5e642f-kube-api-access-8jq5f\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197178 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197106 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-model-cache\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197178 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197128 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b878a7-3169-4390-9184-79f9ff5e642f-tls-certs\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197178 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197165 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-tmp-dir\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197310 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197206 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-home\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197310 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197242 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-dshm\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197524 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197503 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-model-cache\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197616 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197521 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-tmp-dir\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.197616 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.197609 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-home\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.199518 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.199501 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-dshm\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.199809 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.199793 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b878a7-3169-4390-9184-79f9ff5e642f-tls-certs\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.206441 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.206419 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jq5f\" (UniqueName: \"kubernetes.io/projected/d3b878a7-3169-4390-9184-79f9ff5e642f-kube-api-access-8jq5f\") pod \"sub-test-model-7cc2e5-kserve-65db9444d5-dvs58\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.364548 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.364522 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:07:56.493271 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.493087 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58"] Jul 23 12:07:56.496113 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:07:56.496089 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3b878a7_3169_4390_9184_79f9ff5e642f.slice/crio-b272270e53985ade7ccd3cc68a315b51a490e20ef67c2c6072522943ac9ac233 WatchSource:0}: Error finding container b272270e53985ade7ccd3cc68a315b51a490e20ef67c2c6072522943ac9ac233: Status 404 returned error can't find the container with id b272270e53985ade7ccd3cc68a315b51a490e20ef67c2c6072522943ac9ac233 Jul 23 12:07:56.500813 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:56.500789 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e028aa09-a255-4bd5-9756-4edaa721ef7f" path="/var/lib/kubelet/pods/e028aa09-a255-4bd5-9756-4edaa721ef7f/volumes" Jul 23 12:07:57.220961 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:57.220924 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" event={"ID":"d3b878a7-3169-4390-9184-79f9ff5e642f","Type":"ContainerStarted","Data":"a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1"} Jul 23 12:07:57.220961 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:57.220964 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" event={"ID":"d3b878a7-3169-4390-9184-79f9ff5e642f","Type":"ContainerStarted","Data":"b272270e53985ade7ccd3cc68a315b51a490e20ef67c2c6072522943ac9ac233"} Jul 23 12:07:57.238005 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:07:57.237940 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" podStartSLOduration=1.237922595 podStartE2EDuration="1.237922595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:07:57.237307746 +0000 UTC m=+2469.255194392" watchObservedRunningTime="2026-07-23 12:07:57.237922595 +0000 UTC m=+2469.255809232" Jul 23 12:08:06.364892 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:08:06.364854 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:08:06.365304 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:08:06.364908 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:08:06.377251 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:08:06.377221 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:08:07.275862 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:08:07.275831 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:09:20.112525 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:20.112490 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58"] Jul 23 12:09:35.133371 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.133304 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" podUID="d3b878a7-3169-4390-9184-79f9ff5e642f" containerName="main" containerID="cri-o://a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1" gracePeriod=45 Jul 23 12:09:35.380925 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.380898 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:09:35.554703 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.554640 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-model-cache\") pod \"d3b878a7-3169-4390-9184-79f9ff5e642f\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " Jul 23 12:09:35.554703 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.554669 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-dshm\") pod \"d3b878a7-3169-4390-9184-79f9ff5e642f\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " Jul 23 12:09:35.554703 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.554692 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-tmp-dir\") pod \"d3b878a7-3169-4390-9184-79f9ff5e642f\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " Jul 23 12:09:35.554929 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.554713 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jq5f\" (UniqueName: \"kubernetes.io/projected/d3b878a7-3169-4390-9184-79f9ff5e642f-kube-api-access-8jq5f\") pod \"d3b878a7-3169-4390-9184-79f9ff5e642f\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " Jul 23 12:09:35.554929 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.554736 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-home\") pod \"d3b878a7-3169-4390-9184-79f9ff5e642f\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " Jul 23 12:09:35.554929 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.554769 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b878a7-3169-4390-9184-79f9ff5e642f-tls-certs\") pod \"d3b878a7-3169-4390-9184-79f9ff5e642f\" (UID: \"d3b878a7-3169-4390-9184-79f9ff5e642f\") " Jul 23 12:09:35.555080 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.554922 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3b878a7-3169-4390-9184-79f9ff5e642f" (UID: "d3b878a7-3169-4390-9184-79f9ff5e642f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:35.555080 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.554954 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3b878a7-3169-4390-9184-79f9ff5e642f" (UID: "d3b878a7-3169-4390-9184-79f9ff5e642f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:35.555182 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.555073 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-home" (OuterVolumeSpecName: "home") pod "d3b878a7-3169-4390-9184-79f9ff5e642f" (UID: "d3b878a7-3169-4390-9184-79f9ff5e642f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:35.555182 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.555165 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:09:35.555268 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.555209 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:09:35.555268 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.555223 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:09:35.557001 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.556979 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-dshm" (OuterVolumeSpecName: "dshm") pod "d3b878a7-3169-4390-9184-79f9ff5e642f" (UID: "d3b878a7-3169-4390-9184-79f9ff5e642f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:35.557001 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.556985 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b878a7-3169-4390-9184-79f9ff5e642f-kube-api-access-8jq5f" (OuterVolumeSpecName: "kube-api-access-8jq5f") pod "d3b878a7-3169-4390-9184-79f9ff5e642f" (UID: "d3b878a7-3169-4390-9184-79f9ff5e642f"). InnerVolumeSpecName "kube-api-access-8jq5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:09:35.557114 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.557016 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b878a7-3169-4390-9184-79f9ff5e642f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3b878a7-3169-4390-9184-79f9ff5e642f" (UID: "d3b878a7-3169-4390-9184-79f9ff5e642f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:09:35.651207 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.651182 2556 generic.go:356] "Generic (PLEG): container finished" podID="d3b878a7-3169-4390-9184-79f9ff5e642f" containerID="a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1" exitCode=0 Jul 23 12:09:35.651309 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.651223 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" Jul 23 12:09:35.651309 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.651273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" event={"ID":"d3b878a7-3169-4390-9184-79f9ff5e642f","Type":"ContainerDied","Data":"a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1"} Jul 23 12:09:35.651421 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.651317 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58" event={"ID":"d3b878a7-3169-4390-9184-79f9ff5e642f","Type":"ContainerDied","Data":"b272270e53985ade7ccd3cc68a315b51a490e20ef67c2c6072522943ac9ac233"} Jul 23 12:09:35.651421 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.651340 2556 scope.go:117] "RemoveContainer" containerID="a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1" Jul 23 12:09:35.656677 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.656652 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b878a7-3169-4390-9184-79f9ff5e642f-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:09:35.656760 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.656683 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jq5f\" (UniqueName: \"kubernetes.io/projected/d3b878a7-3169-4390-9184-79f9ff5e642f-kube-api-access-8jq5f\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:09:35.656760 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.656698 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b878a7-3169-4390-9184-79f9ff5e642f-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:09:35.659697 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.659686 2556 scope.go:117] "RemoveContainer" containerID="a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1" Jul 23 12:09:35.659944 ip-10-0-129-204 kubenswrapper[2556]: E0723 12:09:35.659927 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1\": container with ID starting with a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1 not found: ID does not exist" containerID="a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1" Jul 23 12:09:35.659991 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.659950 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1"} err="failed to get container status \"a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1\": rpc error: code = NotFound desc = could not find container \"a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1\": container with ID starting with a07830de5688cd35535cdd401916b780b20ed7ff9075485cd923057b1febfda1 not found: ID does not exist" Jul 23 12:09:35.672151 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.672127 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58"] Jul 23 12:09:35.676290 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:35.676272 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7cc2e5/sub-test-model-7cc2e5-kserve-65db9444d5-dvs58"] Jul 23 12:09:36.500676 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:36.500642 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3b878a7-3169-4390-9184-79f9ff5e642f" path="/var/lib/kubelet/pods/d3b878a7-3169-4390-9184-79f9ff5e642f/volumes" Jul 23 12:09:48.624982 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:48.624951 2556 scope.go:117] "RemoveContainer" containerID="eaf8e0d56409504e0124375dcd979bdac1d1952b29007c764ee1af70413d9476" Jul 23 12:09:49.932610 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.932554 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8"] Jul 23 12:09:49.932939 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.932893 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b878a7-3169-4390-9184-79f9ff5e642f" containerName="main" Jul 23 12:09:49.932939 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.932905 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b878a7-3169-4390-9184-79f9ff5e642f" containerName="main" Jul 23 12:09:49.933018 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.932984 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b878a7-3169-4390-9184-79f9ff5e642f" containerName="main" Jul 23 12:09:49.937400 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.937380 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:49.941480 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.941458 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-77f87c\"/\"openshift-service-ca.crt\"" Jul 23 12:09:49.941480 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.941480 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-77f87c\"/\"kube-root-ca.crt\"" Jul 23 12:09:49.941683 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.941459 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-77f87c\"/\"rate-test-model-77f87c-kserve-self-signed-certs\"" Jul 23 12:09:49.941683 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.941459 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-77f87c\"/\"default-dockercfg-xx7rr\"" Jul 23 12:09:49.948272 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:49.948239 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8"] Jul 23 12:09:50.067479 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.067450 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-model-cache\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.067609 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.067483 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-dshm\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.067609 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.067512 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b2dq\" (UniqueName: \"kubernetes.io/projected/80acf233-bf41-42e8-89d8-cc82874b2756-kube-api-access-8b2dq\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.067609 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.067537 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80acf233-bf41-42e8-89d8-cc82874b2756-tls-certs\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.067609 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.067565 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-home\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.067743 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.067640 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-tmp-dir\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.167958 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.167932 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-model-cache\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.168084 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.167963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-dshm\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.168084 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.168001 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b2dq\" (UniqueName: \"kubernetes.io/projected/80acf233-bf41-42e8-89d8-cc82874b2756-kube-api-access-8b2dq\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.168242 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.168115 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80acf233-bf41-42e8-89d8-cc82874b2756-tls-certs\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.168242 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.168166 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-home\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.168362 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.168272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-tmp-dir\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.168419 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.168381 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-model-cache\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.168507 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.168487 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-home\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.168563 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.168503 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-tmp-dir\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.170322 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.170293 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-dshm\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.170650 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.170635 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80acf233-bf41-42e8-89d8-cc82874b2756-tls-certs\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.176969 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.176941 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b2dq\" (UniqueName: \"kubernetes.io/projected/80acf233-bf41-42e8-89d8-cc82874b2756-kube-api-access-8b2dq\") pod \"rate-test-model-77f87c-kserve-5464d7cd75-259w8\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.246323 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.246268 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:09:50.375318 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.375286 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8"] Jul 23 12:09:50.378202 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:09:50.378176 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80acf233_bf41_42e8_89d8_cc82874b2756.slice/crio-0340d6fa42b003fd2b978e031ce245559845b916689f5e5f9e202914aebf4adf WatchSource:0}: Error finding container 0340d6fa42b003fd2b978e031ce245559845b916689f5e5f9e202914aebf4adf: Status 404 returned error can't find the container with id 0340d6fa42b003fd2b978e031ce245559845b916689f5e5f9e202914aebf4adf Jul 23 12:09:50.718897 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.718866 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" event={"ID":"80acf233-bf41-42e8-89d8-cc82874b2756","Type":"ContainerStarted","Data":"435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c"} Jul 23 12:09:50.719074 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.718911 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" event={"ID":"80acf233-bf41-42e8-89d8-cc82874b2756","Type":"ContainerStarted","Data":"0340d6fa42b003fd2b978e031ce245559845b916689f5e5f9e202914aebf4adf"} Jul 23 12:09:50.734713 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:09:50.734661 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" podStartSLOduration=1.734642121 podStartE2EDuration="1.734642121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:09:50.734209476 +0000 UTC m=+2582.752096111" watchObservedRunningTime="2026-07-23 12:09:50.734642121 +0000 UTC m=+2582.752528758" Jul 23 12:10:00.247073 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:10:00.247028 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:10:00.247684 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:10:00.247123 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:10:00.259547 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:10:00.259525 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:10:00.775784 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:10:00.775760 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:10:48.644840 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:10:48.644811 2556 scope.go:117] "RemoveContainer" containerID="07072b9557aaa25fd677f54589cb86354aef380b33843a92427905762687bee8" Jul 23 12:10:49.356471 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:10:49.356427 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8"] Jul 23 12:11:04.377096 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.377030 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" podUID="80acf233-bf41-42e8-89d8-cc82874b2756" containerName="main" containerID="cri-o://435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c" gracePeriod=45 Jul 23 12:11:04.627006 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.626979 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:11:04.645855 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.645780 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-dshm\") pod \"80acf233-bf41-42e8-89d8-cc82874b2756\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " Jul 23 12:11:04.645855 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.645823 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b2dq\" (UniqueName: \"kubernetes.io/projected/80acf233-bf41-42e8-89d8-cc82874b2756-kube-api-access-8b2dq\") pod \"80acf233-bf41-42e8-89d8-cc82874b2756\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " Jul 23 12:11:04.645855 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.645849 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-tmp-dir\") pod \"80acf233-bf41-42e8-89d8-cc82874b2756\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " Jul 23 12:11:04.646086 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.645878 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80acf233-bf41-42e8-89d8-cc82874b2756-tls-certs\") pod \"80acf233-bf41-42e8-89d8-cc82874b2756\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " Jul 23 12:11:04.646086 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.645907 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-model-cache\") pod \"80acf233-bf41-42e8-89d8-cc82874b2756\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " Jul 23 12:11:04.646086 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.646014 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-home\") pod \"80acf233-bf41-42e8-89d8-cc82874b2756\" (UID: \"80acf233-bf41-42e8-89d8-cc82874b2756\") " Jul 23 12:11:04.647302 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.646403 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80acf233-bf41-42e8-89d8-cc82874b2756" (UID: "80acf233-bf41-42e8-89d8-cc82874b2756"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:04.647302 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.646447 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-home" (OuterVolumeSpecName: "home") pod "80acf233-bf41-42e8-89d8-cc82874b2756" (UID: "80acf233-bf41-42e8-89d8-cc82874b2756"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:04.647302 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.646646 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-model-cache" (OuterVolumeSpecName: "model-cache") pod "80acf233-bf41-42e8-89d8-cc82874b2756" (UID: "80acf233-bf41-42e8-89d8-cc82874b2756"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:04.648940 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.648903 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80acf233-bf41-42e8-89d8-cc82874b2756-kube-api-access-8b2dq" (OuterVolumeSpecName: "kube-api-access-8b2dq") pod "80acf233-bf41-42e8-89d8-cc82874b2756" (UID: "80acf233-bf41-42e8-89d8-cc82874b2756"). InnerVolumeSpecName "kube-api-access-8b2dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:04.648940 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.648930 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-dshm" (OuterVolumeSpecName: "dshm") pod "80acf233-bf41-42e8-89d8-cc82874b2756" (UID: "80acf233-bf41-42e8-89d8-cc82874b2756"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:04.649917 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.649895 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80acf233-bf41-42e8-89d8-cc82874b2756-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80acf233-bf41-42e8-89d8-cc82874b2756" (UID: "80acf233-bf41-42e8-89d8-cc82874b2756"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:11:04.746639 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.746617 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:11:04.746639 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.746637 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b2dq\" (UniqueName: \"kubernetes.io/projected/80acf233-bf41-42e8-89d8-cc82874b2756-kube-api-access-8b2dq\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:11:04.746758 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.746648 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:11:04.746758 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.746657 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80acf233-bf41-42e8-89d8-cc82874b2756-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:11:04.746758 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.746665 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:11:04.746758 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:04.746672 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80acf233-bf41-42e8-89d8-cc82874b2756-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:11:05.050892 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.050812 2556 generic.go:356] "Generic (PLEG): container finished" podID="80acf233-bf41-42e8-89d8-cc82874b2756" containerID="435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c" exitCode=0 Jul 23 12:11:05.050892 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.050855 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" Jul 23 12:11:05.051093 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.050896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" event={"ID":"80acf233-bf41-42e8-89d8-cc82874b2756","Type":"ContainerDied","Data":"435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c"} Jul 23 12:11:05.051093 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.050943 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8" event={"ID":"80acf233-bf41-42e8-89d8-cc82874b2756","Type":"ContainerDied","Data":"0340d6fa42b003fd2b978e031ce245559845b916689f5e5f9e202914aebf4adf"} Jul 23 12:11:05.051093 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.050964 2556 scope.go:117] "RemoveContainer" containerID="435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c" Jul 23 12:11:05.059481 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.059464 2556 scope.go:117] "RemoveContainer" containerID="435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c" Jul 23 12:11:05.059734 ip-10-0-129-204 kubenswrapper[2556]: E0723 12:11:05.059716 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c\": container with ID starting with 435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c not found: ID does not exist" containerID="435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c" Jul 23 12:11:05.059798 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.059743 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c"} err="failed to get container status \"435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c\": rpc error: code = NotFound desc = could not find container \"435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c\": container with ID starting with 435a82ff2004800e857f7672c337244367d14bddd864c93beaa28bb49472d48c not found: ID does not exist" Jul 23 12:11:05.071123 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.071098 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8"] Jul 23 12:11:05.077497 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:05.077478 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-77f87c/rate-test-model-77f87c-kserve-5464d7cd75-259w8"] Jul 23 12:11:06.500498 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:06.500464 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80acf233-bf41-42e8-89d8-cc82874b2756" path="/var/lib/kubelet/pods/80acf233-bf41-42e8-89d8-cc82874b2756/volumes" Jul 23 12:11:08.907407 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.907367 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh"] Jul 23 12:11:08.907804 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.907739 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80acf233-bf41-42e8-89d8-cc82874b2756" containerName="main" Jul 23 12:11:08.907804 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.907757 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="80acf233-bf41-42e8-89d8-cc82874b2756" containerName="main" Jul 23 12:11:08.907875 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.907814 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="80acf233-bf41-42e8-89d8-cc82874b2756" containerName="main" Jul 23 12:11:08.912125 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.912106 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" Jul 23 12:11:08.914597 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.914554 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:11:08.914742 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.914605 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 12:11:08.918113 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.918084 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh"] Jul 23 12:11:08.974615 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:08.974584 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40f422-714a-4a58-b2cc-bd30705d19c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh\" (UID: \"ff40f422-714a-4a58-b2cc-bd30705d19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" Jul 23 12:11:09.074967 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:09.074930 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40f422-714a-4a58-b2cc-bd30705d19c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh\" (UID: \"ff40f422-714a-4a58-b2cc-bd30705d19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" Jul 23 12:11:09.075485 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:09.075464 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40f422-714a-4a58-b2cc-bd30705d19c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh\" (UID: \"ff40f422-714a-4a58-b2cc-bd30705d19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" Jul 23 12:11:09.223967 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:09.223908 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" Jul 23 12:11:09.358325 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:09.358292 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh"] Jul 23 12:11:09.361094 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:11:09.361069 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff40f422_714a_4a58_b2cc_bd30705d19c7.slice/crio-f04311373e6433492e7d7949d369561c192b293e4e8e9cec71c100f3ed371e65 WatchSource:0}: Error finding container f04311373e6433492e7d7949d369561c192b293e4e8e9cec71c100f3ed371e65: Status 404 returned error can't find the container with id f04311373e6433492e7d7949d369561c192b293e4e8e9cec71c100f3ed371e65 Jul 23 12:11:10.075242 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:10.075216 2556 generic.go:356] "Generic (PLEG): container finished" podID="ff40f422-714a-4a58-b2cc-bd30705d19c7" containerID="92591b4452481b15e456f5bb8e1f637f06f087944361c78392460c9c4fc6c196" exitCode=0 Jul 23 12:11:10.075551 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:10.075304 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" event={"ID":"ff40f422-714a-4a58-b2cc-bd30705d19c7","Type":"ContainerDied","Data":"92591b4452481b15e456f5bb8e1f637f06f087944361c78392460c9c4fc6c196"} Jul 23 12:11:10.075551 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:10.075357 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" event={"ID":"ff40f422-714a-4a58-b2cc-bd30705d19c7","Type":"ContainerStarted","Data":"f04311373e6433492e7d7949d369561c192b293e4e8e9cec71c100f3ed371e65"} Jul 23 12:11:11.210283 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:11.210259 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" Jul 23 12:11:11.292395 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:11.292364 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40f422-714a-4a58-b2cc-bd30705d19c7-maas-api-service-ca\") pod \"ff40f422-714a-4a58-b2cc-bd30705d19c7\" (UID: \"ff40f422-714a-4a58-b2cc-bd30705d19c7\") " Jul 23 12:11:11.292695 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:11.292675 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff40f422-714a-4a58-b2cc-bd30705d19c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff40f422-714a-4a58-b2cc-bd30705d19c7" (UID: "ff40f422-714a-4a58-b2cc-bd30705d19c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:11:11.393325 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:11.393274 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40f422-714a-4a58-b2cc-bd30705d19c7-maas-api-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:11:12.087057 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:12.087013 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" event={"ID":"ff40f422-714a-4a58-b2cc-bd30705d19c7","Type":"ContainerDied","Data":"f04311373e6433492e7d7949d369561c192b293e4e8e9cec71c100f3ed371e65"} Jul 23 12:11:12.087057 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:12.087032 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh" Jul 23 12:11:12.087057 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:12.087061 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f04311373e6433492e7d7949d369561c192b293e4e8e9cec71c100f3ed371e65" Jul 23 12:11:24.011348 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:24.011310 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh"] Jul 23 12:11:24.013778 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:24.013600 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7cc2e5-5vqxh"] Jul 23 12:11:24.499054 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:24.499024 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff40f422-714a-4a58-b2cc-bd30705d19c7" path="/var/lib/kubelet/pods/ff40f422-714a-4a58-b2cc-bd30705d19c7/volumes" Jul 23 12:11:31.243423 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.243381 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4"] Jul 23 12:11:31.243856 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.243745 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff40f422-714a-4a58-b2cc-bd30705d19c7" containerName="revoke-keys" Jul 23 12:11:31.243856 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.243759 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff40f422-714a-4a58-b2cc-bd30705d19c7" containerName="revoke-keys" Jul 23 12:11:31.243856 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.243819 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff40f422-714a-4a58-b2cc-bd30705d19c7" containerName="revoke-keys" Jul 23 12:11:31.246799 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.246779 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" Jul 23 12:11:31.249717 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.249693 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:11:31.249717 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.249709 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 12:11:31.257788 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.257764 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4"] Jul 23 12:11:31.344366 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.344339 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784ac0d9-af8b-43b2-85fb-64647bcb8ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4\" (UID: \"784ac0d9-af8b-43b2-85fb-64647bcb8ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" Jul 23 12:11:31.445201 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.445170 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784ac0d9-af8b-43b2-85fb-64647bcb8ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4\" (UID: \"784ac0d9-af8b-43b2-85fb-64647bcb8ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" Jul 23 12:11:31.445846 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.445816 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784ac0d9-af8b-43b2-85fb-64647bcb8ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4\" (UID: \"784ac0d9-af8b-43b2-85fb-64647bcb8ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" Jul 23 12:11:31.558757 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.558699 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" Jul 23 12:11:31.688647 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:31.688624 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4"] Jul 23 12:11:31.690684 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:11:31.690650 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784ac0d9_af8b_43b2_85fb_64647bcb8ae3.slice/crio-419e10d700b5c897ee73de4f3ff6ab25b900dc847d206ad35990bf0113ed266c WatchSource:0}: Error finding container 419e10d700b5c897ee73de4f3ff6ab25b900dc847d206ad35990bf0113ed266c: Status 404 returned error can't find the container with id 419e10d700b5c897ee73de4f3ff6ab25b900dc847d206ad35990bf0113ed266c Jul 23 12:11:32.176941 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:32.176866 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" event={"ID":"784ac0d9-af8b-43b2-85fb-64647bcb8ae3","Type":"ContainerStarted","Data":"697cb63d92f66f2aa7fc1e74b923d87f9ed64e8652dbd79191905957814b4ce4"} Jul 23 12:11:32.176941 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:32.176921 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" event={"ID":"784ac0d9-af8b-43b2-85fb-64647bcb8ae3","Type":"ContainerStarted","Data":"419e10d700b5c897ee73de4f3ff6ab25b900dc847d206ad35990bf0113ed266c"} Jul 23 12:11:32.194325 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:32.194281 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" podStartSLOduration=1.194267379 podStartE2EDuration="1.194267379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:11:32.192582552 +0000 UTC m=+2684.210469187" watchObservedRunningTime="2026-07-23 12:11:32.194267379 +0000 UTC m=+2684.212154014" Jul 23 12:11:33.183253 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:33.183221 2556 generic.go:356] "Generic (PLEG): container finished" podID="784ac0d9-af8b-43b2-85fb-64647bcb8ae3" containerID="697cb63d92f66f2aa7fc1e74b923d87f9ed64e8652dbd79191905957814b4ce4" exitCode=0 Jul 23 12:11:33.183639 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:33.183312 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" event={"ID":"784ac0d9-af8b-43b2-85fb-64647bcb8ae3","Type":"ContainerDied","Data":"697cb63d92f66f2aa7fc1e74b923d87f9ed64e8652dbd79191905957814b4ce4"} Jul 23 12:11:34.312208 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:34.312182 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" Jul 23 12:11:34.369795 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:34.369759 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784ac0d9-af8b-43b2-85fb-64647bcb8ae3-maas-api-service-ca\") pod \"784ac0d9-af8b-43b2-85fb-64647bcb8ae3\" (UID: \"784ac0d9-af8b-43b2-85fb-64647bcb8ae3\") " Jul 23 12:11:34.370080 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:34.370059 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784ac0d9-af8b-43b2-85fb-64647bcb8ae3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "784ac0d9-af8b-43b2-85fb-64647bcb8ae3" (UID: "784ac0d9-af8b-43b2-85fb-64647bcb8ae3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:11:34.470698 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:34.470642 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784ac0d9-af8b-43b2-85fb-64647bcb8ae3-maas-api-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\"" Jul 23 12:11:35.193917 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:35.193888 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" Jul 23 12:11:35.194075 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:35.193918 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4" event={"ID":"784ac0d9-af8b-43b2-85fb-64647bcb8ae3","Type":"ContainerDied","Data":"419e10d700b5c897ee73de4f3ff6ab25b900dc847d206ad35990bf0113ed266c"} Jul 23 12:11:35.194075 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:35.193961 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="419e10d700b5c897ee73de4f3ff6ab25b900dc847d206ad35990bf0113ed266c" Jul 23 12:11:46.331672 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:46.331595 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4"] Jul 23 12:11:46.334010 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:46.333984 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-77f87c-9z9w4"] Jul 23 12:11:46.499916 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:46.499883 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784ac0d9-af8b-43b2-85fb-64647bcb8ae3" path="/var/lib/kubelet/pods/784ac0d9-af8b-43b2-85fb-64647bcb8ae3/volumes" Jul 23 12:11:48.567562 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:48.567526 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 12:11:48.570677 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:11:48.570656 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnjnv_2b616b9b-a4ac-4feb-a082-6eae00565d60/ovn-acl-logging/0.log" Jul 23 12:12:00.005418 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:00.005392 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674bfcc66c-xqjnv_d6425183-0989-4a79-913f-94573d32fd36/copy-manifests/0.log" Jul 23 12:12:00.015806 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:00.015780 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674bfcc66c-xqjnv_d6425183-0989-4a79-913f-94573d32fd36/manager/0.log" Jul 23 12:12:03.525850 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:03.525819 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-74x8b_5fbaaa96-64db-4d91-9f11-aa3d25f86695/discovery/0.log" Jul 23 12:12:03.640633 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:03.640601 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766d5cd5c5-25z6p_83f6ed15-a9de-4496-bb25-3064208777f0/kube-auth-proxy/0.log" Jul 23 12:12:04.862185 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:04.862147 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ztbrh_33c8cd5b-7755-4dfd-a4b3-53d1f04ba55b/main/0.log" Jul 23 12:12:04.984793 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:04.984758 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5ws62_08fc63e6-081d-4596-aaed-5fad30417197/main/0.log" Jul 23 12:12:13.674338 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:13.674300 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4bqk7_110693d2-151e-459c-8789-094e101446fc/global-pull-secret-syncer/0.log" Jul 23 12:12:13.865689 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:13.865656 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gznv2_e080c8a4-51cf-4821-bade-e8378e6690ce/konnectivity-agent/0.log" Jul 23 12:12:13.950441 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:13.950349 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-204.ec2.internal_202d9bf1b3a2ad64be78b9941f522638/haproxy/0.log" Jul 23 12:12:21.587267 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:21.587242 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77b8b7dfc8-lbrxx_9dd9983c-a018-43dc-9cae-2466adbfe13d/metrics-server/0.log" Jul 23 12:12:21.615441 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:21.615414 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-wthnq_2407f87a-df88-46b0-aeb8-b23af86a9773/monitoring-plugin/0.log" Jul 23 12:12:21.760028 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:21.759985 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7c725_1a937215-dff4-42e8-a0b7-e85d9f070cb8/node-exporter/0.log" Jul 23 12:12:21.791427 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:21.791400 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7c725_1a937215-dff4-42e8-a0b7-e85d9f070cb8/kube-rbac-proxy/0.log" Jul 23 12:12:21.826207 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:21.826182 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7c725_1a937215-dff4-42e8-a0b7-e85d9f070cb8/init-textfile/0.log" Jul 23 12:12:21.950064 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:21.949994 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-25m4p_ec47f205-a8a4-4a70-8723-dc826e0b439f/kube-rbac-proxy-main/0.log" Jul 23 12:12:21.976966 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:21.976941 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-25m4p_ec47f205-a8a4-4a70-8723-dc826e0b439f/kube-rbac-proxy-self/0.log" Jul 23 12:12:21.999396 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:21.999370 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-25m4p_ec47f205-a8a4-4a70-8723-dc826e0b439f/openshift-state-metrics/0.log" Jul 23 12:12:22.055852 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.055822 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5aaa6bed-6679-427b-805e-bb52ff158a9a/prometheus/0.log" Jul 23 12:12:22.077903 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.077874 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5aaa6bed-6679-427b-805e-bb52ff158a9a/config-reloader/0.log" Jul 23 12:12:22.102868 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.102838 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5aaa6bed-6679-427b-805e-bb52ff158a9a/thanos-sidecar/0.log" Jul 23 12:12:22.129352 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.129327 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5aaa6bed-6679-427b-805e-bb52ff158a9a/kube-rbac-proxy-web/0.log" Jul 23 12:12:22.154938 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.154913 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5aaa6bed-6679-427b-805e-bb52ff158a9a/kube-rbac-proxy/0.log" Jul 23 12:12:22.180281 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.180245 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5aaa6bed-6679-427b-805e-bb52ff158a9a/kube-rbac-proxy-thanos/0.log" Jul 23 12:12:22.208557 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.208484 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5aaa6bed-6679-427b-805e-bb52ff158a9a/init-config-reloader/0.log" Jul 23 12:12:22.243167 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.243133 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6z4q5_a925b66d-6ea3-4a1b-9528-ecda0d251c92/prometheus-operator/0.log" Jul 23 12:12:22.270774 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.270750 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6z4q5_a925b66d-6ea3-4a1b-9528-ecda0d251c92/kube-rbac-proxy/0.log" Jul 23 12:12:22.299439 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.299411 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-g4lwt_0a4cd02f-2573-45e8-a321-5de35dc4c38f/prometheus-operator-admission-webhook/0.log" Jul 23 12:12:22.336707 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.336681 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c9ffd4bcc-5cvz4_7d79be3c-e42d-4cad-97f5-c0b88a503494/telemeter-client/0.log" Jul 23 12:12:22.367508 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.367478 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c9ffd4bcc-5cvz4_7d79be3c-e42d-4cad-97f5-c0b88a503494/reload/0.log" Jul 23 12:12:22.395335 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:22.395309 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c9ffd4bcc-5cvz4_7d79be3c-e42d-4cad-97f5-c0b88a503494/kube-rbac-proxy/0.log" Jul 23 12:12:23.831049 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.831008 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p"] Jul 23 12:12:23.831436 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.831332 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784ac0d9-af8b-43b2-85fb-64647bcb8ae3" containerName="revoke-keys" Jul 23 12:12:23.831436 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.831343 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="784ac0d9-af8b-43b2-85fb-64647bcb8ae3" containerName="revoke-keys" Jul 23 12:12:23.831436 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.831410 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="784ac0d9-af8b-43b2-85fb-64647bcb8ae3" containerName="revoke-keys" Jul 23 12:12:23.834883 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.834857 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.837218 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.837196 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xw2jd\"/\"kube-root-ca.crt\"" Jul 23 12:12:23.838142 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.838117 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xw2jd\"/\"default-dockercfg-k7f76\"" Jul 23 12:12:23.838259 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.838164 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xw2jd\"/\"openshift-service-ca.crt\"" Jul 23 12:12:23.845723 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.845696 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p"] Jul 23 12:12:23.897359 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.897314 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-lib-modules\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.897359 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.897356 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-podres\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.897624 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.897378 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-proc\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.897624 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.897478 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-sys\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.897624 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.897519 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cgg6\" (UniqueName: \"kubernetes.io/projected/095b1ce7-0364-488b-b9b9-c3519c185f1f-kube-api-access-4cgg6\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.998450 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998407 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-proc\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.998691 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998483 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-sys\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.998691 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cgg6\" (UniqueName: \"kubernetes.io/projected/095b1ce7-0364-488b-b9b9-c3519c185f1f-kube-api-access-4cgg6\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.998691 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998542 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-proc\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.998691 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998556 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-lib-modules\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.998691 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998625 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-podres\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.998691 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998639 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-sys\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.999011 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998716 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-lib-modules\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:23.999011 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:23.998736 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/095b1ce7-0364-488b-b9b9-c3519c185f1f-podres\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:24.007391 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:24.007359 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cgg6\" (UniqueName: \"kubernetes.io/projected/095b1ce7-0364-488b-b9b9-c3519c185f1f-kube-api-access-4cgg6\") pod \"perf-node-gather-daemonset-l5w4p\" (UID: \"095b1ce7-0364-488b-b9b9-c3519c185f1f\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:24.146117 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:24.146007 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:24.286475 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:24.286445 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p"] Jul 23 12:12:24.288653 ip-10-0-129-204 kubenswrapper[2556]: W0723 12:12:24.288624 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod095b1ce7_0364_488b_b9b9_c3519c185f1f.slice/crio-24e448098ec95b279c656b132ff85dac11b26c5d29ef754d36c7b2dedb911957 WatchSource:0}: Error finding container 24e448098ec95b279c656b132ff85dac11b26c5d29ef754d36c7b2dedb911957: Status 404 returned error can't find the container with id 24e448098ec95b279c656b132ff85dac11b26c5d29ef754d36c7b2dedb911957 Jul 23 12:12:24.412551 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:24.412454 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" event={"ID":"095b1ce7-0364-488b-b9b9-c3519c185f1f","Type":"ContainerStarted","Data":"2935c463339a40debdd234b936f49354a2ce5e42c8457e0141070999d38c233f"} Jul 23 12:12:24.412551 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:24.412506 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" event={"ID":"095b1ce7-0364-488b-b9b9-c3519c185f1f","Type":"ContainerStarted","Data":"24e448098ec95b279c656b132ff85dac11b26c5d29ef754d36c7b2dedb911957"} Jul 23 12:12:24.412761 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:24.412618 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:24.429958 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:24.429895 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" podStartSLOduration=1.429873495 podStartE2EDuration="1.429873495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 12:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:12:24.429631681 +0000 UTC m=+2736.447518318" watchObservedRunningTime="2026-07-23 12:12:24.429873495 +0000 UTC m=+2736.447760131" Jul 23 12:12:26.838136 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:26.838110 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8q59n_1ca94b96-042b-4db5-af4d-9855aec04c28/dns/0.log" Jul 23 12:12:26.886130 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:26.886099 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8q59n_1ca94b96-042b-4db5-af4d-9855aec04c28/kube-rbac-proxy/0.log" Jul 23 12:12:27.135516 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:27.135440 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxgc5_c9dde853-d0e2-40da-bb68-ab7f67f456a7/dns-node-resolver/0.log" Jul 23 12:12:27.866114 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:27.866074 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r5v5q_19c37710-4dfc-4e47-82dc-3622c2ace92c/node-ca/0.log" Jul 23 12:12:28.976482 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:28.976451 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-74x8b_5fbaaa96-64db-4d91-9f11-aa3d25f86695/discovery/0.log" Jul 23 12:12:29.000993 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:29.000962 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766d5cd5c5-25z6p_83f6ed15-a9de-4496-bb25-3064208777f0/kube-auth-proxy/0.log" Jul 23 12:12:29.967880 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:29.967847 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fvjmm_83c4d15b-687e-487a-8f65-12945db48eac/serve-healthcheck-canary/0.log" Jul 23 12:12:30.428027 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:30.427995 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-l5w4p" Jul 23 12:12:30.542715 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:30.542690 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q8j9_8580b584-f0d6-439a-b6d7-7e6d29ae82be/kube-rbac-proxy/0.log" Jul 23 12:12:30.574165 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:30.574135 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q8j9_8580b584-f0d6-439a-b6d7-7e6d29ae82be/exporter/0.log" Jul 23 12:12:30.601354 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:30.601328 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q8j9_8580b584-f0d6-439a-b6d7-7e6d29ae82be/extractor/0.log" Jul 23 12:12:33.938080 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:33.938045 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674bfcc66c-xqjnv_d6425183-0989-4a79-913f-94573d32fd36/manager/0.log" Jul 23 12:12:33.964455 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:33.964427 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674bfcc66c-xqjnv_d6425183-0989-4a79-913f-94573d32fd36/copy-manifests/0.log" Jul 23 12:12:34.511878 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:34.511846 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-74x8b_5fbaaa96-64db-4d91-9f11-aa3d25f86695/discovery/0.log" Jul 23 12:12:34.522300 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:34.522272 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766d5cd5c5-25z6p_83f6ed15-a9de-4496-bb25-3064208777f0/kube-auth-proxy/0.log" Jul 23 12:12:35.323740 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:35.323711 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c45f5478c-jfttb_f836c371-db0e-4210-8874-0b1d6ba04522/manager/0.log" Jul 23 12:12:36.004221 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:36.004187 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-74x8b_5fbaaa96-64db-4d91-9f11-aa3d25f86695/discovery/0.log" Jul 23 12:12:36.023690 ip-10-0-129-204 kubenswrapper[2556]: I0723 12:12:36.023663 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766d5cd5c5-25z6p_83f6ed15-a9de-4496-bb25-3064208777f0/kube-auth-proxy/0.log"