Jul 21 13:44:25.649235 ip-10-0-129-45 systemd[1]: Starting Kubernetes Kubelet... Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.977491 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980798 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980811 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980815 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980818 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980821 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980824 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980827 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 13:44:26.101777 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980829 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 13:44:26.039132 ip-10-0-129-45 systemd[1]: Started Kubernetes Kubelet. Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980832 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980835 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980838 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980842 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980845 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980847 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980850 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980853 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980856 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980860 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980863 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980866 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980869 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980872 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980875 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980878 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980881 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980884 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980888 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 13:44:26.103476 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980891 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980894 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980896 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980899 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980901 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980904 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980906 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980909 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980914 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980918 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980920 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980923 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980926 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980929 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980931 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980934 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980936 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980939 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980942 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 13:44:26.105951 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980945 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980947 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980950 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980952 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980955 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980972 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980975 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980977 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980980 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980983 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980985 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980988 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980992 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980995 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.980997 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981000 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981002 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981005 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981008 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981011 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 13:44:26.108016 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981013 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981017 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981020 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981022 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981025 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981029 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981032 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981035 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981037 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981040 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981042 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981045 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981048 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981051 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981054 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981056 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981059 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981062 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981065 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 13:44:26.110069 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981067 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981070 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981072 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981075 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981078 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981080 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981083 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981085 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981088 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981091 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981093 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981097 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981100 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981103 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981785 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981791 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981795 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981798 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981801 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981805 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 13:44:26.112497 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981808 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981810 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981813 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981815 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981818 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981820 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981822 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981825 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981828 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981831 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981833 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981836 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981839 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981841 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981844 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981846 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981848 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981851 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981853 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981856 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 13:44:26.113768 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981858 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981861 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981863 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981866 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981868 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981871 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981874 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981876 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981879 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981882 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981884 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981886 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981889 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981892 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981895 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981897 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981899 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981902 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981904 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981907 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 13:44:26.114949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981910 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981913 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981917 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981920 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981923 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981926 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981929 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981932 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981935 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981940 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981943 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981946 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981949 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981951 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981954 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981977 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981980 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981982 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981985 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 13:44:26.116127 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981988 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981990 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981993 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981996 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.981999 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982002 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982005 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982008 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982010 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982013 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982016 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982018 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982021 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982023 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982026 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982029 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982031 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982034 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982038 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982040 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 13:44:26.116905 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982043 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982045 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982048 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982051 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982054 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982057 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982059 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982062 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982065 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982069 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982071 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982074 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982076 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982079 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982499 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982505 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982508 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982511 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982515 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982518 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 13:44:26.117680 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982521 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982524 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982526 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982529 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982531 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982534 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982536 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982539 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982543 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982546 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982548 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982551 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982554 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982557 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982559 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982563 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982565 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982568 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982571 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 13:44:26.118774 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982573 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982576 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982578 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982584 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982586 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982589 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982591 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982595 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982597 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982600 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982603 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982605 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982609 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982611 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982614 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982617 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982620 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982622 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982625 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982627 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 13:44:26.119551 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982630 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982632 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982635 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982637 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982640 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982643 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982645 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982648 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982650 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982653 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982655 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982658 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982660 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982662 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982665 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982669 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982672 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982674 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982677 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 13:44:26.120317 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982681 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982684 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982687 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982690 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982693 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982696 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982699 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982701 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982704 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982707 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982709 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982711 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982714 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982716 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982719 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982723 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982727 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982731 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982735 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982739 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 13:44:26.121081 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982742 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982744 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982747 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982749 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982752 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982754 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982757 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982760 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982764 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982766 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982769 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982772 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982774 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982776 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.982780 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982878 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982886 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982892 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982897 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982902 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 13:44:26.121846 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982905 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982909 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982913 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982917 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982920 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982924 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982930 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982933 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982936 2569 flags.go:64] FLAG: --cgroup-root="" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982939 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982942 2569 flags.go:64] FLAG: --client-ca-file="" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982944 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982947 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982952 2569 flags.go:64] FLAG: --cluster-domain="" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982955 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982970 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982974 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982978 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982981 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982985 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982989 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982992 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982995 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.982998 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 13:44:26.122628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983001 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983004 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983007 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983010 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983013 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983016 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983019 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983022 2569 flags.go:64] FLAG: --enable-server="true" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983025 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983029 2569 flags.go:64] FLAG: --event-burst="100" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983032 2569 flags.go:64] FLAG: --event-qps="50" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983035 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983038 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983041 2569 flags.go:64] FLAG: --eviction-hard="" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983047 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983050 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983053 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983056 2569 flags.go:64] FLAG: --eviction-soft="" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983058 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983061 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983064 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983067 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983070 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983073 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983075 2569 flags.go:64] FLAG: --feature-gates="" Jul 21 13:44:26.123803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983079 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983082 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983085 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983089 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983092 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983095 2569 flags.go:64] FLAG: --help="false" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983098 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-129-45.ec2.internal" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983101 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983104 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983107 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983110 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983113 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983116 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983119 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983122 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983125 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983129 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983132 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983135 2569 flags.go:64] FLAG: --kube-reserved="" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983138 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983141 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983145 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983148 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983150 2569 flags.go:64] FLAG: --lock-file="" Jul 21 13:44:26.124970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983153 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983156 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983159 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983165 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983167 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983170 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983173 2569 flags.go:64] FLAG: --logging-format="text" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983176 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983179 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983182 2569 flags.go:64] FLAG: --manifest-url="" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983185 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983190 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983193 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983197 2569 flags.go:64] FLAG: --max-pods="110" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983200 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983203 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983206 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983209 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983212 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983215 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983218 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983224 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983227 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983230 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 13:44:26.126138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983233 2569 flags.go:64] FLAG: --pod-cidr="" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983237 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983243 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983246 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983249 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983253 2569 flags.go:64] FLAG: --port="10250" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983256 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983259 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-089750e7294eb9816" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983262 2569 flags.go:64] FLAG: --qos-reserved="" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983265 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983268 2569 flags.go:64] FLAG: --register-node="true" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983270 2569 flags.go:64] FLAG: --register-with-taints="" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983274 2569 flags.go:64] FLAG: --registry-burst="10" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983277 2569 flags.go:64] FLAG: --registry-qps="5" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983279 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983282 2569 flags.go:64] FLAG: --reserved-memory="" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983285 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983288 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983291 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983294 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983298 2569 flags.go:64] FLAG: --runonce="false" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983301 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983304 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983307 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983310 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 13:44:26.130788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983313 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983316 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983319 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983321 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983324 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983327 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983330 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983333 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983336 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983339 2569 flags.go:64] FLAG: --system-cgroups="" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983342 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983347 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983351 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983354 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983358 2569 flags.go:64] FLAG: --tls-min-version="" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983361 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983364 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983367 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983369 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983372 2569 flags.go:64] FLAG: --v="2" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983376 2569 flags.go:64] FLAG: --version="false" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983379 2569 flags.go:64] FLAG: --vmodule="" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983383 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.983386 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983511 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 13:44:26.133656 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983515 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983519 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983524 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983527 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983530 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983533 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983535 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983538 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983541 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983543 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983546 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983548 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983551 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983554 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983556 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983559 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983562 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983565 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983567 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983572 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 13:44:26.166949 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983574 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983577 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983579 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983582 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983584 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983587 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983589 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983592 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983594 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983597 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983600 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983602 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983604 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983607 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983610 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983613 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983616 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983619 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983621 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983624 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 13:44:26.167720 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983627 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983629 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983632 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983634 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983637 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983639 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983642 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983644 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983647 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983649 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983652 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983656 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983658 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983661 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983663 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983666 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983668 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983671 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983673 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 13:44:26.168510 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983676 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983678 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983681 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983683 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983686 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983688 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983691 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983696 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983699 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983702 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983704 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983707 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983710 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983712 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983715 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983717 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983720 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983722 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983725 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 13:44:26.169295 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983727 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983730 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983732 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983735 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983737 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983745 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983748 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983750 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983753 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983755 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983758 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983760 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983763 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983766 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983770 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983773 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983776 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983778 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983781 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 13:44:26.170067 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.983784 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.984386 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.990732 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.990746 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.990759 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.990764 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990819 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990824 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990829 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990832 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990836 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990839 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990842 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990845 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 13:44:26.170819 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990848 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990851 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990853 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990856 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990860 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990862 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990865 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990867 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990871 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990873 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990876 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990879 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990881 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990886 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990889 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990892 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990896 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990899 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990902 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 13:44:26.171589 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990905 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990907 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990910 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990913 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990916 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990918 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990921 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990923 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990926 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990928 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990931 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990933 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990936 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990939 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990941 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990944 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990946 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990949 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990953 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 13:44:26.172362 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990956 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990977 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990982 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990985 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990988 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990990 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990993 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.990998 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991001 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991003 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991006 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991009 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991012 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991014 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991017 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991019 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991023 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991025 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991028 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991031 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 13:44:26.173134 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991033 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991036 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991038 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991041 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991043 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991046 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991048 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991051 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991054 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991056 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991059 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991062 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991065 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991067 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991070 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991072 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991075 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991077 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991080 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991085 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 13:44:26.266219 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991087 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991090 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991092 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991095 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991098 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991101 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991103 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991106 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991108 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991111 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991115 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991118 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991121 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.991126 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991233 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 13:44:26.266918 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991238 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991241 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991243 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991246 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991248 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991251 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991253 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991256 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991258 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991260 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991267 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991270 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991273 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991275 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991278 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991280 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991284 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991288 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991290 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991293 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 13:44:26.267684 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991295 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991298 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991301 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991304 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991306 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991309 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991313 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991317 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991320 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991323 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991326 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991329 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991331 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991334 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991337 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991340 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991343 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991345 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991348 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 13:44:26.268443 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991350 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991353 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991356 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991358 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991362 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991366 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991369 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991371 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991374 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991379 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991383 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991387 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991389 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991392 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991395 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991398 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991401 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991403 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991406 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 13:44:26.269207 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991408 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991411 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991414 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991418 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991421 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991423 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991426 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991429 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991431 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991433 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991436 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991438 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991441 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991443 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991446 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991448 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991451 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991453 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991456 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991459 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 13:44:26.269986 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991461 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991464 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991468 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991471 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991474 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991476 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991479 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991481 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991484 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991487 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991489 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991492 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991494 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991497 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991499 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991502 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991505 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991507 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991510 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 13:44:26.270739 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:25.991512 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.991517 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.991612 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.993508 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.994655 2569 server.go:1019] "Starting client certificate rotation" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.994761 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.994829 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:25.996765 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.000511 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.008313 2569 log.go:25] "Validated CRI v1 runtime API" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.011555 2569 log.go:25] "Validated CRI v1 image API" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.012347 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 13:44:26.271488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.014813 2569 fs.go:135] Filesystem UUIDs: map[79cd8ef5-7703-4161-8012-1d52cc7e0ec1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f85aae89-0cf7-4acc-bdfa-115cf1477f21:/dev/nvme0n1p3] Jul 21 13:44:26.271877 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.014827 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 13:44:26.272269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.019470 2569 manager.go:216] Machine: {Timestamp:2026-07-21 13:44:26.0184365 +0000 UTC m=+0.276084834 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099150 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a67f3e0ccab4782bc2e31e1f01828 SystemUUID:ec2a67f3-e0cc-ab47-82bc-2e31e1f01828 BootID:9a56fad4-0121-4103-bc32-711e9e0d8412 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:28:5a:9c:3c:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:28:5a:9c:3c:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:c2:84:a6:72:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 13:44:26.272269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.019570 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 13:44:26.272269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.019676 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 13:44:26.272269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.020606 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 13:44:26.272269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.020627 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 13:44:26.272269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.021255 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 13:44:26.272269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.021268 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 13:44:26.272269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.021276 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.021857 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.023228 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.023479 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027154 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027272 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027300 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027310 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027325 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027336 2569 kubelet.go:398] "Adding apiserver pod source" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027345 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027671 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.027726 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.029688 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.030912 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.030939 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031750 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031763 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031769 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031774 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031782 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031787 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031795 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031800 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031807 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031812 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031820 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.031825 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.032356 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.032364 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.035604 2569 server.go:1268] "Started kubelet" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.038411 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.038481 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.035688 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.040311 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.040997 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.041543 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.041713 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.041610 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 13:44:26.272646 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.045342 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.045093 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-45.ec2.internal.18c451f53a7b9626 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-45.ec2.internal,UID:ip-10-0-129-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-45.ec2.internal,},FirstTimestamp:2026-07-21 13:44:26.03557431 +0000 UTC m=+0.293222639,LastTimestamp:2026-07-21 13:44:26.03557431 +0000 UTC m=+0.293222639,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-45.ec2.internal,}" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.048361 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79mwq" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.050114 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.050123 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.050187 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.050595 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.050599 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.050614 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.050749 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.050756 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.051233 2569 factory.go:55] Registering systemd factory Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.051247 2569 factory.go:223] Registration of the systemd container factory successfully Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.051524 2569 factory.go:153] Registering CRI-O factory Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.051539 2569 factory.go:223] Registration of the crio container factory successfully Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.051599 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.051618 2569 factory.go:103] Registering Raw factory Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.051627 2569 manager.go:1196] Started watching for new ooms in manager Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.051906 2569 manager.go:349] Starting recovery of all containers Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.052861 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.052945 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.056146 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79mwq" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.061099 2569 manager.go:354] Recovery completed Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.065835 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.067631 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.067655 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.067665 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.068149 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.068159 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.068175 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.068177 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.068766 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.068789 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 13:44:26.277291 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.068798 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID" Jul 21 13:44:26.278435 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.070208 2569 policy_none.go:49] "None policy: Start" Jul 21 13:44:26.278435 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.070219 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 13:44:26.278435 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.070229 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 13:44:26.278435 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:26.072211 2569 policy_none.go:47] "Start" Jul 21 13:44:26.278435 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.151160 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:26.278435 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.251387 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:26.383377 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.351766 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:26.496709 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.452140 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:26.600448 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:26.552682 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:27.925480 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:27.872574 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.035096 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:27.972931 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.142683 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.073350 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.249244 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.173739 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.249244 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.213884 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 13:44:28.249244 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.214083 2569 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae368d438ac564eff893aa625eeb48c1-83c6bffc42408f95.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-45.ec2.internal.18c451f53c64e711\": read tcp 10.0.129.45:58772->54.88.219.128:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-45.ec2.internal.18c451f53c64e711 default 6302 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-45.ec2.internal,UID:ip-10-0-129-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-45.ec2.internal,},FirstTimestamp:2026-07-21 13:44:26 +0000 UTC,LastTimestamp:2026-07-21 13:44:26.068776648 +0000 UTC m=+0.326424977,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-45.ec2.internal,}" Jul 21 13:44:28.367608 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.274360 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.367608 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.281499 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 13:39:27 +0000 UTC" deadline="2028-02-29 19:15:20.639438376 +0000 UTC" Jul 21 13:44:28.367608 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.281522 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14117h30m52.357920072s" Jul 21 13:44:28.378062 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.375343 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.484139 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.475761 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.586198 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.576101 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.692277 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.676432 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.715650 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.715627 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 13:44:28.723757 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.723736 2569 manager.go:337] "Starting Device Plugin manager" Jul 21 13:44:28.723872 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.723780 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 13:44:28.723872 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.723792 2569 server.go:87] "Starting device plugin registration server" Jul 21 13:44:28.724078 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.724066 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 13:44:28.724124 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.724080 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 13:44:28.724233 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.724210 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 13:44:28.724381 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.724312 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 13:44:28.724381 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.724325 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 13:44:28.724907 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.724888 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 13:44:28.724986 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.724943 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-45.ec2.internal\" not found" Jul 21 13:44:28.731670 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.731655 2569 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-45.ec2.internal" not found Jul 21 13:44:28.760907 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.760891 2569 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-45.ec2.internal" not found Jul 21 13:44:28.778318 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.778302 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 13:44:28.787972 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.787944 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 13:44:28.789108 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.789088 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 13:44:28.789169 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.789123 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 13:44:28.789169 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.789141 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 13:44:28.789263 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:28.789170 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 13:44:28.791674 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.791654 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 13:44:28.819791 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.819768 2569 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-45.ec2.internal" not found Jul 21 13:44:28.824870 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.824846 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 13:44:28.825745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.825730 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory" Jul 21 13:44:28.825815 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.825764 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 13:44:28.825815 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.825780 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID" Jul 21 13:44:28.825815 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.825810 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-45.ec2.internal" Jul 21 13:44:28.832803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.832789 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-45.ec2.internal" Jul 21 13:44:28.890175 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.890108 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"] Jul 21 13:44:28.892580 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.892564 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" Jul 21 13:44:28.894613 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.894596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:28.913056 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.913041 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" Jul 21 13:44:28.916349 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.916336 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:28.934591 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.934572 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 13:44:28.934665 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.934608 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 13:44:28.978826 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.978804 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa01869be574e582ebebcc2a4c69c415-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"fa01869be574e582ebebcc2a4c69c415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:28.978910 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.978828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa01869be574e582ebebcc2a4c69c415-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"fa01869be574e582ebebcc2a4c69c415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:28.978910 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:28.978846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b9b4eb6aa309c78e5ae7c65fd70512d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"3b9b4eb6aa309c78e5ae7c65fd70512d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.079429 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.079404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa01869be574e582ebebcc2a4c69c415-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"fa01869be574e582ebebcc2a4c69c415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.079573 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.079434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa01869be574e582ebebcc2a4c69c415-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"fa01869be574e582ebebcc2a4c69c415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.079573 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.079459 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b9b4eb6aa309c78e5ae7c65fd70512d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"3b9b4eb6aa309c78e5ae7c65fd70512d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.079573 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.079509 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa01869be574e582ebebcc2a4c69c415-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"fa01869be574e582ebebcc2a4c69c415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.079573 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.079561 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b9b4eb6aa309c78e5ae7c65fd70512d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"3b9b4eb6aa309c78e5ae7c65fd70512d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.079722 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.079509 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa01869be574e582ebebcc2a4c69c415-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"fa01869be574e582ebebcc2a4c69c415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.220991 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.220902 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 13:44:29.238496 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.238467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.239564 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.239535 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" Jul 21 13:44:29.248094 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.248078 2569 apiserver.go:52] "Watching apiserver" Jul 21 13:44:29.256589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.256565 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 13:44:29.257511 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.257492 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ztpvd","kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977","openshift-cluster-node-tuning-operator/tuned-26m6q","openshift-dns/node-resolver-5gg2v","openshift-image-registry/node-ca-tfwjv","openshift-multus/multus-fhbbq","kube-system/konnectivity-agent-ktcmb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal","openshift-multus/multus-additional-cni-plugins-grzqj","openshift-multus/network-metrics-daemon-drjp7","openshift-network-diagnostics/network-check-target-r9rll","openshift-network-operator/iptables-alerter-dn4cz","openshift-ovn-kubernetes/ovnkube-node-zzfr5"] Jul 21 13:44:29.260069 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.260050 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.262241 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.262217 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.262848 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.262830 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 13:44:29.262982 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.262836 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l7495\"" Jul 21 13:44:29.263066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.262836 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 13:44:29.263098 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.262862 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 13:44:29.263130 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.262927 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 13:44:29.264379 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.264359 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 13:44:29.264532 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.264514 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.264687 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.264668 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 13:44:29.264755 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.264701 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wlqmd\"" Jul 21 13:44:29.264926 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.264906 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 13:44:29.266796 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.266781 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.267425 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.267411 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 13:44:29.268994 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.268975 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.269254 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.269233 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 13:44:29.269450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.269435 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 13:44:29.269570 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.269554 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 13:44:29.269629 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.269608 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 13:44:29.269629 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.269623 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkh97\"" Jul 21 13:44:29.269726 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.269608 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 13:44:29.270306 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.270284 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 13:44:29.270710 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.270689 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h8zq8\"" Jul 21 13:44:29.270801 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.270711 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 13:44:29.270801 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.270721 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 13:44:29.271029 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.271012 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.271204 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.271187 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gjlmz\"" Jul 21 13:44:29.272483 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.272459 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 13:44:29.272642 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.272623 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 13:44:29.272991 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.272933 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-77mnp\"" Jul 21 13:44:29.273259 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.273243 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 13:44:29.273345 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.273315 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 13:44:29.273607 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.273593 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 13:44:29.274177 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.274155 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:29.276313 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.276295 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hxvbv\"" Jul 21 13:44:29.276386 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.276352 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 13:44:29.276623 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.276606 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 13:44:29.276946 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.276930 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.279210 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.279013 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-255sv\"" Jul 21 13:44:29.279381 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.279369 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:29.279565 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.279512 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:29.279665 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.279646 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 13:44:29.279723 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.279566 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 13:44:29.280590 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-systemd\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.280650 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280607 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c088994-9870-4c82-9f4e-25e99e2c348d-serviceca\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.280650 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280628 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-cni-bin\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.280744 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-run\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.280744 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280670 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6nc\" (UniqueName: \"kubernetes.io/projected/30c50200-8929-4ef8-b57c-79638beebc58-kube-api-access-kg6nc\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.280744 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280715 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-etc-kubernetes\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.280887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f88f8987-bcbb-4091-8f8e-70f14f7e766d-iptables-alerter-script\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.280887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280782 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-system-cni-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.280887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-k8s-cni-cncf-io\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.280887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-cni-multus\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.280887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280853 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-daemon-config\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.280887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-etc-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281084 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280898 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysconfig\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.281084 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-kubelet\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281084 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281084 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.280992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54a0d21e-e20a-480d-b43a-ffcf50d30691-tmp\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.281084 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ad1ea75-3b12-4393-9140-9b290d946dff-cni-binary-copy\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281084 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281044 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-kubelet\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281084 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-lib-modules\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-host\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281100 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c088994-9870-4c82-9f4e-25e99e2c348d-host\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtpfw\" (UniqueName: \"kubernetes.io/projected/6c088994-9870-4c82-9f4e-25e99e2c348d-kube-api-access-gtpfw\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-os-release\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281179 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1847b46-65d9-4412-b09c-931245edd671-ovn-node-metrics-cert\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysctl-conf\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281211 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-cnibin\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281232 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-node-log\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-cni-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-slash\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-cni-netd\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-ovnkube-script-lib\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-conf-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-hostroot\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281397 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-sys\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-var-lib-kubelet\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c50200-8929-4ef8-b57c-79638beebc58-tmp-dir\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-netns\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281505 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-ovn\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-cni-bin\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281570 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-env-overrides\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281602 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkk7\" (UniqueName: \"kubernetes.io/projected/a1847b46-65d9-4412-b09c-931245edd671-kube-api-access-zdkk7\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.281715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281628 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-modprobe-d\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n48v\" (UniqueName: \"kubernetes.io/projected/6ad1ea75-3b12-4393-9140-9b290d946dff-kube-api-access-4n48v\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-run-netns\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-kubernetes\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281735 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysctl-d\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281757 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-tuned\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-socket-dir-parent\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-systemd-units\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkcm\" (UniqueName: \"kubernetes.io/projected/54a0d21e-e20a-480d-b43a-ffcf50d30691-kube-api-access-xtkcm\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281847 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281849 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30c50200-8929-4ef8-b57c-79638beebc58-hosts-file\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.281911 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281931 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f88f8987-bcbb-4091-8f8e-70f14f7e766d-host-slash\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbq8w\" (UniqueName: \"kubernetes.io/projected/f88f8987-bcbb-4091-8f8e-70f14f7e766d-kube-api-access-lbq8w\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.282186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-multus-certs\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.282601 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281983 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-var-lib-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.282601 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.281996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-log-socket\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.282601 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.282012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-ovnkube-config\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.282601 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.282028 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-systemd\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.284160 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.284145 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.284217 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.284202 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:29.286374 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.286356 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.288369 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.288351 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 13:44:29.288485 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.288468 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 13:44:29.288550 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.288496 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 13:44:29.288822 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.288799 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 13:44:29.288894 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.288854 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xh5b9\"" Jul 21 13:44:29.316300 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.316281 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f7z5g" Jul 21 13:44:29.322183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.322167 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f7z5g" Jul 21 13:44:29.371840 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.371825 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 13:44:29.382199 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-cni-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.382272 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382205 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-slash\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382272 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382272 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-cni-netd\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382272 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-ovnkube-script-lib\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382272 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382266 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-sys-fs\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-conf-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382297 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382340 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-slash\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382352 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-registration-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382319 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-cni-netd\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382378 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-hostroot\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382390 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-conf-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382424 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382424 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-sys\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-hostroot\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-cni-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-var-lib-kubelet\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.382461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382460 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-sys\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382484 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c50200-8929-4ef8-b57c-79638beebc58-tmp-dir\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382492 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-var-lib-kubelet\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cni-binary-copy\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382556 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-netns\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-ovn\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382587 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-netns\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-kubelet-config\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382616 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-ovn\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-cni-bin\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-env-overrides\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382675 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-cni-bin\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkk7\" (UniqueName: \"kubernetes.io/projected/a1847b46-65d9-4412-b09c-931245edd671-kube-api-access-zdkk7\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382730 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-modprobe-d\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382754 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382782 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n48v\" (UniqueName: \"kubernetes.io/projected/6ad1ea75-3b12-4393-9140-9b290d946dff-kube-api-access-4n48v\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-run-netns\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382831 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-kubernetes\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-modprobe-d\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382847 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-ovnkube-script-lib\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382851 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c50200-8929-4ef8-b57c-79638beebc58-tmp-dir\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysctl-d\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-tuned\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382907 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-run-netns\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382932 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-kubernetes\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382933 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cnibin\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.382990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383009 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysctl-d\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.383780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383012 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-env-overrides\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383034 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9lt\" (UniqueName: \"kubernetes.io/projected/bf8a6a44-03aa-440b-bb95-ca157db9024c-kube-api-access-2r9lt\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-socket-dir-parent\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-systemd-units\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383115 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkcm\" (UniqueName: \"kubernetes.io/projected/54a0d21e-e20a-480d-b43a-ffcf50d30691-kube-api-access-xtkcm\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383126 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-socket-dir-parent\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383134 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-systemd-units\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383141 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30c50200-8929-4ef8-b57c-79638beebc58-hosts-file\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383173 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-system-cni-dir\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383185 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30c50200-8929-4ef8-b57c-79638beebc58-hosts-file\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383198 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383226 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383243 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f88f8987-bcbb-4091-8f8e-70f14f7e766d-host-slash\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.384284 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383274 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbq8w\" (UniqueName: \"kubernetes.io/projected/f88f8987-bcbb-4091-8f8e-70f14f7e766d-kube-api-access-lbq8w\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-multus-certs\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383316 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f88f8987-bcbb-4091-8f8e-70f14f7e766d-host-slash\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-var-lib-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-log-socket\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383363 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-multus-certs\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-ovnkube-config\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383376 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-var-lib-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383393 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h522z\" (UniqueName: \"kubernetes.io/projected/68d0bf37-60ca-48b7-80f0-87d86a9858d8-kube-api-access-h522z\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383409 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-log-socket\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383461 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-systemd\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bb9j\" (UniqueName: \"kubernetes.io/projected/7bdb30ab-7639-4ed5-9d22-354a94d7b791-kube-api-access-7bb9j\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383504 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-dbus\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383507 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-run-systemd\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.384923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383533 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-systemd\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c088994-9870-4c82-9f4e-25e99e2c348d-serviceca\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383590 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-systemd\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383594 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-cni-bin\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383634 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-cni-bin\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383661 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-run\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383693 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6nc\" (UniqueName: \"kubernetes.io/projected/30c50200-8929-4ef8-b57c-79638beebc58-kube-api-access-kg6nc\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383717 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383754 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-run\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-etc-kubernetes\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f88f8987-bcbb-4091-8f8e-70f14f7e766d-iptables-alerter-script\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-system-cni-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383897 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1847b46-65d9-4412-b09c-931245edd671-ovnkube-config\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-k8s-cni-cncf-io\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383957 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-etc-kubernetes\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.385588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383962 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-system-cni-dir\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383972 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c088994-9870-4c82-9f4e-25e99e2c348d-serviceca\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-cni-multus\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.383999 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-run-k8s-cni-cncf-io\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-daemon-config\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384037 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-cni-multus\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384038 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-etc-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysconfig\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384069 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-etc-openvswitch\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-kubelet\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384116 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysconfig\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384117 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384147 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-os-release\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384160 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.386243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f018ddca-f4a1-444c-b602-f0a34a3f3269-konnectivity-ca\") pod \"konnectivity-agent-ktcmb\" (UID: \"f018ddca-f4a1-444c-b602-f0a34a3f3269\") " pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384177 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-host-var-lib-kubelet\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54a0d21e-e20a-480d-b43a-ffcf50d30691-tmp\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384239 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ad1ea75-3b12-4393-9140-9b290d946dff-cni-binary-copy\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-kubelet\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-lib-modules\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-host\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c088994-9870-4c82-9f4e-25e99e2c348d-host\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384348 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtpfw\" (UniqueName: \"kubernetes.io/projected/6c088994-9870-4c82-9f4e-25e99e2c348d-kube-api-access-gtpfw\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384371 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f018ddca-f4a1-444c-b602-f0a34a3f3269-agent-certs\") pod \"konnectivity-agent-ktcmb\" (UID: \"f018ddca-f4a1-444c-b602-f0a34a3f3269\") " pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-os-release\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384417 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1847b46-65d9-4412-b09c-931245edd671-ovn-node-metrics-cert\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysctl-conf\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-device-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.386865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384488 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-cnibin\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-node-log\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-socket-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384599 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ad1ea75-3b12-4393-9140-9b290d946dff-multus-daemon-config\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384603 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f88f8987-bcbb-4091-8f8e-70f14f7e766d-iptables-alerter-script\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384670 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-os-release\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384704 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c088994-9870-4c82-9f4e-25e99e2c348d-host\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384716 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ad1ea75-3b12-4393-9140-9b290d946dff-cni-binary-copy\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384722 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ad1ea75-3b12-4393-9140-9b290d946dff-cnibin\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384749 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-host\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384765 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-host-kubelet\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384766 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1847b46-65d9-4412-b09c-931245edd671-node-log\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384845 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-lib-modules\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.384852 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-sysctl-conf\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.387341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.386895 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54a0d21e-e20a-480d-b43a-ffcf50d30691-etc-tuned\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.387760 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.386916 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54a0d21e-e20a-480d-b43a-ffcf50d30691-tmp\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.387760 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.386994 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1847b46-65d9-4412-b09c-931245edd671-ovn-node-metrics-cert\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.396731 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.396705 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbq8w\" (UniqueName: \"kubernetes.io/projected/f88f8987-bcbb-4091-8f8e-70f14f7e766d-kube-api-access-lbq8w\") pod \"iptables-alerter-dn4cz\" (UID: \"f88f8987-bcbb-4091-8f8e-70f14f7e766d\") " pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.396973 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.396943 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkcm\" (UniqueName: \"kubernetes.io/projected/54a0d21e-e20a-480d-b43a-ffcf50d30691-kube-api-access-xtkcm\") pod \"tuned-26m6q\" (UID: \"54a0d21e-e20a-480d-b43a-ffcf50d30691\") " pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.397156 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.397134 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n48v\" (UniqueName: \"kubernetes.io/projected/6ad1ea75-3b12-4393-9140-9b290d946dff-kube-api-access-4n48v\") pod \"multus-fhbbq\" (UID: \"6ad1ea75-3b12-4393-9140-9b290d946dff\") " pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.397351 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.397335 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtpfw\" (UniqueName: \"kubernetes.io/projected/6c088994-9870-4c82-9f4e-25e99e2c348d-kube-api-access-gtpfw\") pod \"node-ca-tfwjv\" (UID: \"6c088994-9870-4c82-9f4e-25e99e2c348d\") " pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.397616 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.397601 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkk7\" (UniqueName: \"kubernetes.io/projected/a1847b46-65d9-4412-b09c-931245edd671-kube-api-access-zdkk7\") pod \"ovnkube-node-zzfr5\" (UID: \"a1847b46-65d9-4412-b09c-931245edd671\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.397822 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.397808 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6nc\" (UniqueName: \"kubernetes.io/projected/30c50200-8929-4ef8-b57c-79638beebc58-kube-api-access-kg6nc\") pod \"node-resolver-5gg2v\" (UID: \"30c50200-8929-4ef8-b57c-79638beebc58\") " pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.484770 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-os-release\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484803 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f018ddca-f4a1-444c-b602-f0a34a3f3269-konnectivity-ca\") pod \"konnectivity-agent-ktcmb\" (UID: \"f018ddca-f4a1-444c-b602-f0a34a3f3269\") " pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f018ddca-f4a1-444c-b602-f0a34a3f3269-agent-certs\") pod \"konnectivity-agent-ktcmb\" (UID: \"f018ddca-f4a1-444c-b602-f0a34a3f3269\") " pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484894 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-os-release\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-device-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484945 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-device-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484953 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-socket-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.484963 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.484959 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484980 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485008 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-sys-fs\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.485041 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret podName:79c9e89d-d287-46a8-b4fe-eeb2e8a31223 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:29.985005806 +0000 UTC m=+3.023075708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret") pod "global-pull-secret-syncer-ztpvd" (UID: "79c9e89d-d287-46a8-b4fe-eeb2e8a31223") : object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485054 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-sys-fs\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.484960 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-registration-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485121 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485128 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-socket-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cni-binary-copy\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485176 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-registration-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485184 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-kubelet-config\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.485375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485222 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-kubelet-config\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cnibin\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485296 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9lt\" (UniqueName: \"kubernetes.io/projected/bf8a6a44-03aa-440b-bb95-ca157db9024c-kube-api-access-2r9lt\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485302 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cnibin\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-system-cni-dir\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485347 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485428 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h522z\" (UniqueName: \"kubernetes.io/projected/68d0bf37-60ca-48b7-80f0-87d86a9858d8-kube-api-access-h522z\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485453 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdb30ab-7639-4ed5-9d22-354a94d7b791-system-cni-dir\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bb9j\" (UniqueName: \"kubernetes.io/projected/7bdb30ab-7639-4ed5-9d22-354a94d7b791-kube-api-access-7bb9j\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.485496 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:29.485920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485498 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-dbus\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.486263 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.485574 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs podName:68d0bf37-60ca-48b7-80f0-87d86a9858d8 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:29.985536 +0000 UTC m=+3.023605900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs") pod "network-metrics-daemon-drjp7" (UID: "68d0bf37-60ca-48b7-80f0-87d86a9858d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:29.486263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485578 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8a6a44-03aa-440b-bb95-ca157db9024c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.486263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485566 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-dbus\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.486263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485429 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f018ddca-f4a1-444c-b602-f0a34a3f3269-konnectivity-ca\") pod \"konnectivity-agent-ktcmb\" (UID: \"f018ddca-f4a1-444c-b602-f0a34a3f3269\") " pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:29.486263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485655 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cni-binary-copy\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.486263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.485745 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.486263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.486152 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bdb30ab-7639-4ed5-9d22-354a94d7b791-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.487377 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.487361 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f018ddca-f4a1-444c-b602-f0a34a3f3269-agent-certs\") pod \"konnectivity-agent-ktcmb\" (UID: \"f018ddca-f4a1-444c-b602-f0a34a3f3269\") " pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:29.495074 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.495050 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9lt\" (UniqueName: \"kubernetes.io/projected/bf8a6a44-03aa-440b-bb95-ca157db9024c-kube-api-access-2r9lt\") pod \"aws-ebs-csi-driver-node-hs977\" (UID: \"bf8a6a44-03aa-440b-bb95-ca157db9024c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.497855 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.497838 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 13:44:29.497855 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.497857 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 13:44:29.498000 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.497866 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8c9t5 for pod openshift-network-diagnostics/network-check-target-r9rll: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:29.498000 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.497910 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5 podName:c859f694-6549-4863-8b28-9e97b2b52063 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:29.997899784 +0000 UTC m=+3.035969684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9t5" (UniqueName: "kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5") pod "network-check-target-r9rll" (UID: "c859f694-6549-4863-8b28-9e97b2b52063") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:29.499888 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.499869 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bb9j\" (UniqueName: \"kubernetes.io/projected/7bdb30ab-7639-4ed5-9d22-354a94d7b791-kube-api-access-7bb9j\") pod \"multus-additional-cni-plugins-grzqj\" (UID: \"7bdb30ab-7639-4ed5-9d22-354a94d7b791\") " pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.501909 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.501893 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h522z\" (UniqueName: \"kubernetes.io/projected/68d0bf37-60ca-48b7-80f0-87d86a9858d8-kube-api-access-h522z\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:29.589440 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.589413 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fhbbq" Jul 21 13:44:29.602619 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.602599 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dn4cz" Jul 21 13:44:29.618203 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.618187 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:29.636792 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.636767 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26m6q" Jul 21 13:44:29.654297 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.654276 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5gg2v" Jul 21 13:44:29.667321 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.667299 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tfwjv" Jul 21 13:44:29.686253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.686234 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:29.712796 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.712773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-grzqj" Jul 21 13:44:29.715326 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.715305 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" Jul 21 13:44:29.816903 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:29.816861 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9b4eb6aa309c78e5ae7c65fd70512d.slice/crio-c9c10155049a4c27e9ea7f5fdcea4f6c2590107e054f9ec9b0d4beeaa56f15a4 WatchSource:0}: Error finding container c9c10155049a4c27e9ea7f5fdcea4f6c2590107e054f9ec9b0d4beeaa56f15a4: Status 404 returned error can't find the container with id c9c10155049a4c27e9ea7f5fdcea4f6c2590107e054f9ec9b0d4beeaa56f15a4 Jul 21 13:44:29.823959 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.823943 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 13:44:29.828707 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:29.828681 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa01869be574e582ebebcc2a4c69c415.slice/crio-dce7784c2b55f237257ca25e735000b22dce8ccb175a86801bb700078323b441 WatchSource:0}: Error finding container dce7784c2b55f237257ca25e735000b22dce8ccb175a86801bb700078323b441: Status 404 returned error can't find the container with id dce7784c2b55f237257ca25e735000b22dce8ccb175a86801bb700078323b441 Jul 21 13:44:29.945724 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:29.945701 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad1ea75_3b12_4393_9140_9b290d946dff.slice/crio-7a1977211f96113b8ff4ae744968159ff9a2e54f1f57b44bf8881296f2a3c400 WatchSource:0}: Error finding container 7a1977211f96113b8ff4ae744968159ff9a2e54f1f57b44bf8881296f2a3c400: Status 404 returned error can't find the container with id 7a1977211f96113b8ff4ae744968159ff9a2e54f1f57b44bf8881296f2a3c400 Jul 21 13:44:29.989786 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.989759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:29.989871 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:29.989814 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:29.989917 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.989896 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:29.989955 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.989911 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:29.989955 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.989939 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs podName:68d0bf37-60ca-48b7-80f0-87d86a9858d8 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:30.989924581 +0000 UTC m=+4.027994480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs") pod "network-metrics-daemon-drjp7" (UID: "68d0bf37-60ca-48b7-80f0-87d86a9858d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:29.990031 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:29.989967 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret podName:79c9e89d-d287-46a8-b4fe-eeb2e8a31223 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:30.989951237 +0000 UTC m=+4.028021144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret") pod "global-pull-secret-syncer-ztpvd" (UID: "79c9e89d-d287-46a8-b4fe-eeb2e8a31223") : object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:30.007458 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:30.007439 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88f8987_bcbb_4091_8f8e_70f14f7e766d.slice/crio-4bf5acc32e089512ca76686da83d040bc21581ca1a8809eb63b61e8f82707eb0 WatchSource:0}: Error finding container 4bf5acc32e089512ca76686da83d040bc21581ca1a8809eb63b61e8f82707eb0: Status 404 returned error can't find the container with id 4bf5acc32e089512ca76686da83d040bc21581ca1a8809eb63b61e8f82707eb0 Jul 21 13:44:30.085817 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:30.085787 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a0d21e_e20a_480d_b43a_ffcf50d30691.slice/crio-c183b926c0fc1f8e8876d0d37e869745aac27b3fc51e60d8f38ad79ccb32ef4b WatchSource:0}: Error finding container c183b926c0fc1f8e8876d0d37e869745aac27b3fc51e60d8f38ad79ccb32ef4b: Status 404 returned error can't find the container with id c183b926c0fc1f8e8876d0d37e869745aac27b3fc51e60d8f38ad79ccb32ef4b Jul 21 13:44:30.090289 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.090249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:30.090408 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.090393 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 13:44:30.090446 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.090413 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 13:44:30.090446 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.090422 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8c9t5 for pod openshift-network-diagnostics/network-check-target-r9rll: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:30.090575 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.090462 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5 podName:c859f694-6549-4863-8b28-9e97b2b52063 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:31.090449465 +0000 UTC m=+4.128519368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9t5" (UniqueName: "kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5") pod "network-check-target-r9rll" (UID: "c859f694-6549-4863-8b28-9e97b2b52063") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:30.095172 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:30.095149 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf018ddca_f4a1_444c_b602_f0a34a3f3269.slice/crio-b1a53adfa361a64165e618ac1f14d0469872370499dbcef3de088e9e50546206 WatchSource:0}: Error finding container b1a53adfa361a64165e618ac1f14d0469872370499dbcef3de088e9e50546206: Status 404 returned error can't find the container with id b1a53adfa361a64165e618ac1f14d0469872370499dbcef3de088e9e50546206 Jul 21 13:44:30.124132 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:30.124107 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c50200_8929_4ef8_b57c_79638beebc58.slice/crio-7d97e2f437c57f4b0e64b95d819c4487573a640f8d6c4225e858cad489fbd66b WatchSource:0}: Error finding container 7d97e2f437c57f4b0e64b95d819c4487573a640f8d6c4225e858cad489fbd66b: Status 404 returned error can't find the container with id 7d97e2f437c57f4b0e64b95d819c4487573a640f8d6c4225e858cad489fbd66b Jul 21 13:44:30.153409 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:30.153386 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1847b46_65d9_4412_b09c_931245edd671.slice/crio-fb05e03c66a8ad582695557a048a39eed5ba6ab7c04af14b1a4b03edcf672531 WatchSource:0}: Error finding container fb05e03c66a8ad582695557a048a39eed5ba6ab7c04af14b1a4b03edcf672531: Status 404 returned error can't find the container with id fb05e03c66a8ad582695557a048a39eed5ba6ab7c04af14b1a4b03edcf672531 Jul 21 13:44:30.284694 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:30.284664 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c088994_9870_4c82_9f4e_25e99e2c348d.slice/crio-3b3238a4f99182b483c319e617bd3cdcb1dd5b574600c63d428f6d46fdd36751 WatchSource:0}: Error finding container 3b3238a4f99182b483c319e617bd3cdcb1dd5b574600c63d428f6d46fdd36751: Status 404 returned error can't find the container with id 3b3238a4f99182b483c319e617bd3cdcb1dd5b574600c63d428f6d46fdd36751 Jul 21 13:44:30.325186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.325129 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 13:39:29 +0000 UTC" deadline="2028-04-12 02:45:05.869559312 +0000 UTC" Jul 21 13:44:30.325186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.325178 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15133h0m35.544383555s" Jul 21 13:44:30.450773 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:30.450732 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8a6a44_03aa_440b_bb95_ca157db9024c.slice/crio-06c172c3e8780e3a59b7bd19784bca5f2592a291e7b5259849df1706715cae21 WatchSource:0}: Error finding container 06c172c3e8780e3a59b7bd19784bca5f2592a291e7b5259849df1706715cae21: Status 404 returned error can't find the container with id 06c172c3e8780e3a59b7bd19784bca5f2592a291e7b5259849df1706715cae21 Jul 21 13:44:30.577708 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:44:30.577674 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdb30ab_7639_4ed5_9d22_354a94d7b791.slice/crio-e1c4c4dcff5a203eac1dac86c609d7712429dc9c51fde902d172774f6a22f201 WatchSource:0}: Error finding container e1c4c4dcff5a203eac1dac86c609d7712429dc9c51fde902d172774f6a22f201: Status 404 returned error can't find the container with id e1c4c4dcff5a203eac1dac86c609d7712429dc9c51fde902d172774f6a22f201 Jul 21 13:44:30.792417 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.792381 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:30.792616 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.792513 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:30.792683 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.792626 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:30.792747 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.792715 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:30.792803 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.792768 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:30.792854 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.792831 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:30.803152 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.803097 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fhbbq" event={"ID":"6ad1ea75-3b12-4393-9140-9b290d946dff","Type":"ContainerStarted","Data":"7a1977211f96113b8ff4ae744968159ff9a2e54f1f57b44bf8881296f2a3c400"} Jul 21 13:44:30.809372 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.809342 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"3b9b4eb6aa309c78e5ae7c65fd70512d","Type":"ContainerStarted","Data":"c9c10155049a4c27e9ea7f5fdcea4f6c2590107e054f9ec9b0d4beeaa56f15a4"} Jul 21 13:44:30.814728 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.814700 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5gg2v" event={"ID":"30c50200-8929-4ef8-b57c-79638beebc58","Type":"ContainerStarted","Data":"7d97e2f437c57f4b0e64b95d819c4487573a640f8d6c4225e858cad489fbd66b"} Jul 21 13:44:30.821839 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.821806 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktcmb" event={"ID":"f018ddca-f4a1-444c-b602-f0a34a3f3269","Type":"ContainerStarted","Data":"b1a53adfa361a64165e618ac1f14d0469872370499dbcef3de088e9e50546206"} Jul 21 13:44:30.826879 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.826851 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"fa01869be574e582ebebcc2a4c69c415","Type":"ContainerStarted","Data":"dce7784c2b55f237257ca25e735000b22dce8ccb175a86801bb700078323b441"} Jul 21 13:44:30.830235 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.830208 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerStarted","Data":"e1c4c4dcff5a203eac1dac86c609d7712429dc9c51fde902d172774f6a22f201"} Jul 21 13:44:30.833005 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.832957 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" event={"ID":"bf8a6a44-03aa-440b-bb95-ca157db9024c","Type":"ContainerStarted","Data":"06c172c3e8780e3a59b7bd19784bca5f2592a291e7b5259849df1706715cae21"} Jul 21 13:44:30.834920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.834894 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tfwjv" event={"ID":"6c088994-9870-4c82-9f4e-25e99e2c348d","Type":"ContainerStarted","Data":"3b3238a4f99182b483c319e617bd3cdcb1dd5b574600c63d428f6d46fdd36751"} Jul 21 13:44:30.843706 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.843679 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"fb05e03c66a8ad582695557a048a39eed5ba6ab7c04af14b1a4b03edcf672531"} Jul 21 13:44:30.849013 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.848987 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26m6q" event={"ID":"54a0d21e-e20a-480d-b43a-ffcf50d30691","Type":"ContainerStarted","Data":"c183b926c0fc1f8e8876d0d37e869745aac27b3fc51e60d8f38ad79ccb32ef4b"} Jul 21 13:44:30.852316 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.852289 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dn4cz" event={"ID":"f88f8987-bcbb-4091-8f8e-70f14f7e766d","Type":"ContainerStarted","Data":"4bf5acc32e089512ca76686da83d040bc21581ca1a8809eb63b61e8f82707eb0"} Jul 21 13:44:30.997370 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.997329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:30.997594 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:30.997423 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:30.997594 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.997505 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:30.997594 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.997519 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:30.997594 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.997583 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs podName:68d0bf37-60ca-48b7-80f0-87d86a9858d8 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:32.997569189 +0000 UTC m=+6.035639088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs") pod "network-metrics-daemon-drjp7" (UID: "68d0bf37-60ca-48b7-80f0-87d86a9858d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:30.997594 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:30.997596 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret podName:79c9e89d-d287-46a8-b4fe-eeb2e8a31223 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:32.997590342 +0000 UTC m=+6.035660241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret") pod "global-pull-secret-syncer-ztpvd" (UID: "79c9e89d-d287-46a8-b4fe-eeb2e8a31223") : object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:31.098976 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:31.098427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:31.098976 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:31.098602 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 13:44:31.098976 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:31.098629 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 13:44:31.098976 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:31.098642 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8c9t5 for pod openshift-network-diagnostics/network-check-target-r9rll: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:31.098976 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:31.098700 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5 podName:c859f694-6549-4863-8b28-9e97b2b52063 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:33.098681991 +0000 UTC m=+6.136751894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9t5" (UniqueName: "kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5") pod "network-check-target-r9rll" (UID: "c859f694-6549-4863-8b28-9e97b2b52063") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:31.326174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:31.326131 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 13:39:29 +0000 UTC" deadline="2027-12-27 14:42:12.512514701 +0000 UTC" Jul 21 13:44:31.326174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:31.326171 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12576h57m41.186348624s" Jul 21 13:44:32.790661 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:32.790632 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:32.791122 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:32.790774 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:32.791122 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:32.790833 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:32.791122 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:32.790911 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:32.792917 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:32.792761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:32.792917 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:32.792862 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:33.012093 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:33.011692 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:33.012093 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:33.011776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:33.012093 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:33.011902 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:33.012093 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:33.011961 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs podName:68d0bf37-60ca-48b7-80f0-87d86a9858d8 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:37.011942677 +0000 UTC m=+10.050012582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs") pod "network-metrics-daemon-drjp7" (UID: "68d0bf37-60ca-48b7-80f0-87d86a9858d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:33.012093 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:33.012022 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:33.012093 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:33.012061 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret podName:79c9e89d-d287-46a8-b4fe-eeb2e8a31223 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:37.012042179 +0000 UTC m=+10.050112084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret") pod "global-pull-secret-syncer-ztpvd" (UID: "79c9e89d-d287-46a8-b4fe-eeb2e8a31223") : object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:33.113363 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:33.113135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:33.113363 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:33.113289 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 13:44:33.113363 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:33.113310 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 13:44:33.113363 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:33.113323 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8c9t5 for pod openshift-network-diagnostics/network-check-target-r9rll: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:33.113667 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:33.113384 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5 podName:c859f694-6549-4863-8b28-9e97b2b52063 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:37.113366128 +0000 UTC m=+10.151436032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9t5" (UniqueName: "kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5") pod "network-check-target-r9rll" (UID: "c859f694-6549-4863-8b28-9e97b2b52063") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:34.789740 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:34.789657 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:34.790231 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:34.789654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:34.790231 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:34.789804 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:34.790231 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:34.789665 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:34.790231 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:34.789892 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:34.790231 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:34.789986 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:36.790312 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:36.790279 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:36.790808 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:36.790411 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:36.790808 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:36.790476 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:36.790808 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:36.790582 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:36.790808 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:36.790629 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:36.790808 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:36.790698 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:37.045283 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:37.045198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:37.045283 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:37.045265 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:37.045511 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:37.045387 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:37.045511 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:37.045449 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret podName:79c9e89d-d287-46a8-b4fe-eeb2e8a31223 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:45.045430741 +0000 UTC m=+18.083500644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret") pod "global-pull-secret-syncer-ztpvd" (UID: "79c9e89d-d287-46a8-b4fe-eeb2e8a31223") : object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:37.045511 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:37.045511 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:37.045739 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:37.045561 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs podName:68d0bf37-60ca-48b7-80f0-87d86a9858d8 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:45.045530661 +0000 UTC m=+18.083600566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs") pod "network-metrics-daemon-drjp7" (UID: "68d0bf37-60ca-48b7-80f0-87d86a9858d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:37.146035 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:37.145890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:37.146211 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:37.146075 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 13:44:37.146211 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:37.146093 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 13:44:37.146211 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:37.146104 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8c9t5 for pod openshift-network-diagnostics/network-check-target-r9rll: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:37.146211 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:37.146159 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5 podName:c859f694-6549-4863-8b28-9e97b2b52063 nodeName:}" failed. No retries permitted until 2026-07-21 13:44:45.146141902 +0000 UTC m=+18.184211816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9t5" (UniqueName: "kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5") pod "network-check-target-r9rll" (UID: "c859f694-6549-4863-8b28-9e97b2b52063") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:38.790532 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:38.790325 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:38.790998 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:38.790607 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:38.790998 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:38.790683 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:38.790998 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:38.790788 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:38.790998 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:38.790835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:38.790998 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:38.790897 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:40.789640 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:40.789596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:40.789640 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:40.789631 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:40.790126 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:40.789600 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:40.790126 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:40.789727 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:40.790126 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:40.789853 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:40.790126 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:40.789975 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:42.789404 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:42.789370 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:42.789404 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:42.789389 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:42.789845 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:42.789380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:42.789845 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:42.789503 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:42.789845 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:42.789604 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:42.789845 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:42.789695 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:44.789591 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:44.789564 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:44.790005 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:44.789564 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:44.790005 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:44.789681 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:44.790005 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:44.789780 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:44.790005 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:44.789569 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:44.790005 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:44.789908 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:45.109523 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:45.109440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:45.109691 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:45.109526 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:45.109691 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:45.109617 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:45.109691 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:45.109634 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:45.109691 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:45.109687 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs podName:68d0bf37-60ca-48b7-80f0-87d86a9858d8 nodeName:}" failed. No retries permitted until 2026-07-21 13:45:01.109669929 +0000 UTC m=+34.147739829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs") pod "network-metrics-daemon-drjp7" (UID: "68d0bf37-60ca-48b7-80f0-87d86a9858d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:44:45.109857 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:45.109705 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret podName:79c9e89d-d287-46a8-b4fe-eeb2e8a31223 nodeName:}" failed. No retries permitted until 2026-07-21 13:45:01.109697313 +0000 UTC m=+34.147767212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret") pod "global-pull-secret-syncer-ztpvd" (UID: "79c9e89d-d287-46a8-b4fe-eeb2e8a31223") : object "kube-system"/"original-pull-secret" not registered Jul 21 13:44:45.210892 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:45.210857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:45.211072 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:45.211040 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 13:44:45.211072 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:45.211063 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 13:44:45.211186 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:45.211075 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8c9t5 for pod openshift-network-diagnostics/network-check-target-r9rll: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:45.211186 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:45.211136 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5 podName:c859f694-6549-4863-8b28-9e97b2b52063 nodeName:}" failed. No retries permitted until 2026-07-21 13:45:01.211116987 +0000 UTC m=+34.249186900 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9t5" (UniqueName: "kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5") pod "network-check-target-r9rll" (UID: "c859f694-6549-4863-8b28-9e97b2b52063") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:44:46.789753 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:46.789716 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:46.790197 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:46.789729 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:46.790197 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:46.789838 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:46.790197 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:46.789733 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:46.790197 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:46.789909 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:46.790197 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:46.789986 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:48.790582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:48.790523 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:48.791047 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:48.790667 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:48.791047 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:48.790699 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:48.791047 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:48.790800 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:48.791047 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:48.790852 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:48.791047 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:48.790951 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:49.895402 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.894944 2569 generic.go:356] "Generic (PLEG): container finished" podID="7bdb30ab-7639-4ed5-9d22-354a94d7b791" containerID="d3a6378d9427cbb4eff48939e4e04a672c47f8678aa5fb3b2062a6a829f61b81" exitCode=0 Jul 21 13:44:49.896202 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.895039 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerDied","Data":"d3a6378d9427cbb4eff48939e4e04a672c47f8678aa5fb3b2062a6a829f61b81"} Jul 21 13:44:49.897060 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.897040 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" event={"ID":"bf8a6a44-03aa-440b-bb95-ca157db9024c","Type":"ContainerStarted","Data":"2902a6435bf34a0b7c046a40294352278c45ee22d90429ff3cdd8e95ebdb19fc"} Jul 21 13:44:49.898376 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.898357 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tfwjv" event={"ID":"6c088994-9870-4c82-9f4e-25e99e2c348d","Type":"ContainerStarted","Data":"d8e4363718aacf26ba67aaea305570709c6fdee95e2e55d35c6efacbd4ff56d9"} Jul 21 13:44:49.900946 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.900920 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"de520360638bb6a1bdc344dcb7a8eea52961952e9a13bca777f987839184b6b8"} Jul 21 13:44:49.901026 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.900989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"2fa6ba04ad54b93638bdb6bd648c21ed1736e380be5baeef56399a0e546427e0"} Jul 21 13:44:49.901075 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.901040 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"c6db2f9ba009acef7694e55fd470c90c0eafdabdb41fe28ceb9fc4d105d200eb"} Jul 21 13:44:49.901118 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.901087 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"0ae6e49390d273d95748bfb7639370b197fd03e54bea214fbd77236c4961d113"} Jul 21 13:44:49.901168 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.901132 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"cba399bd1b4d3eddc69030d071ca0dc84d3481a24350b92a0dabd8932ecb27b1"} Jul 21 13:44:49.902470 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.902354 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26m6q" event={"ID":"54a0d21e-e20a-480d-b43a-ffcf50d30691","Type":"ContainerStarted","Data":"1e4f58a607eae80737b6e26a3732768e4dee48b7a839dc67cc168564a89e4ec4"} Jul 21 13:44:49.903761 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.903729 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fhbbq" event={"ID":"6ad1ea75-3b12-4393-9140-9b290d946dff","Type":"ContainerStarted","Data":"94ee6e89117f8d365bdccfc6dd234e542ed815b5f597b3cd561384761419394f"} Jul 21 13:44:49.905100 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.905069 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"3b9b4eb6aa309c78e5ae7c65fd70512d","Type":"ContainerStarted","Data":"de9e8dd3af6577831d00ef7adacaa944df31d1bd81678960e2245ec62e53b998"} Jul 21 13:44:49.906320 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.906278 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5gg2v" event={"ID":"30c50200-8929-4ef8-b57c-79638beebc58","Type":"ContainerStarted","Data":"0a7f9f39e8c242719dd5be7f73e06f3e46423ce716fe4507fd74c1eccdbdaa16"} Jul 21 13:44:49.907517 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.907496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktcmb" event={"ID":"f018ddca-f4a1-444c-b602-f0a34a3f3269","Type":"ContainerStarted","Data":"30e7d0285d060fcb61a1e2c1890bc6cad85856ff1af5ecc09afff2e59e4d686a"} Jul 21 13:44:49.908677 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.908659 2569 generic.go:356] "Generic (PLEG): container finished" podID="fa01869be574e582ebebcc2a4c69c415" containerID="97753febde1dbb8440453e569eedf5fb87e736a279be2e33eed26fcec098a068" exitCode=0 Jul 21 13:44:49.908753 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.908683 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"fa01869be574e582ebebcc2a4c69c415","Type":"ContainerDied","Data":"97753febde1dbb8440453e569eedf5fb87e736a279be2e33eed26fcec098a068"} Jul 21 13:44:49.947866 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:49.947827 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tfwjv" podStartSLOduration=8.178626333 podStartE2EDuration="21.94781421s" totalImagesPullingTime="13.769187877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:49.94772238 +0000 UTC m=+22.985792301" watchObservedRunningTime="2026-07-21 13:44:49.94781421 +0000 UTC m=+22.985884130" Jul 21 13:44:50.004152 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.004110 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-26m6q" podStartSLOduration=2.791381584 podStartE2EDuration="22.004095587s" totalImagesPullingTime="19.212714003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:50.003248203 +0000 UTC m=+23.041318121" watchObservedRunningTime="2026-07-21 13:44:50.004095587 +0000 UTC m=+23.042165509" Jul 21 13:44:50.004337 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.004319 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ktcmb" podStartSLOduration=11.753028711 podStartE2EDuration="22.004313474s" totalImagesPullingTime="10.251284763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:49.966041655 +0000 UTC m=+23.004111577" watchObservedRunningTime="2026-07-21 13:44:50.004313474 +0000 UTC m=+23.042383394" Jul 21 13:44:50.030341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.030299 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" podStartSLOduration=22.030285713 podStartE2EDuration="22.030285713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:50.029979857 +0000 UTC m=+23.068049777" watchObservedRunningTime="2026-07-21 13:44:50.030285713 +0000 UTC m=+23.068355634" Jul 21 13:44:50.102300 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.102243 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fhbbq" podStartSLOduration=2.745402556 podStartE2EDuration="22.10222324s" totalImagesPullingTime="19.356820684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:50.101840181 +0000 UTC m=+23.139910104" watchObservedRunningTime="2026-07-21 13:44:50.10222324 +0000 UTC m=+23.140293163" Jul 21 13:44:50.789970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.789946 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:50.789970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.789965 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:50.790207 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.789944 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:50.790207 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:50.790049 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:50.790207 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:50.790101 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:50.790207 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:50.790174 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:50.800817 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.800791 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 13:44:50.912768 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.912696 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" event={"ID":"bf8a6a44-03aa-440b-bb95-ca157db9024c","Type":"ContainerStarted","Data":"595d9a9e7a6cec901554ecbecb203bdd9568bba110b10895285f842b8a64fb4c"} Jul 21 13:44:50.915493 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.915470 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"8cd955f2945d9071f5081a1f9cf7284b7fecafc96df8d7f4e8014844e23acf0a"} Jul 21 13:44:50.916875 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.916852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dn4cz" event={"ID":"f88f8987-bcbb-4091-8f8e-70f14f7e766d","Type":"ContainerStarted","Data":"7718addd11879d9da428f804fe0d32a88d44d437b5102171a1ef4e0d1342879f"} Jul 21 13:44:50.918583 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.918550 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"fa01869be574e582ebebcc2a4c69c415","Type":"ContainerStarted","Data":"f8727e8dfb644bf936dd404fdf2d786edcd9dfd61f8f65e801799164181b3641"} Jul 21 13:44:50.935593 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.935556 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dn4cz" podStartSLOduration=12.596482956 podStartE2EDuration="22.935528607s" totalImagesPullingTime="10.339045651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:50.935367811 +0000 UTC m=+23.973437732" watchObservedRunningTime="2026-07-21 13:44:50.935528607 +0000 UTC m=+23.973598510" Jul 21 13:44:50.935878 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:50.935859 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5gg2v" podStartSLOduration=3.757841556 podStartE2EDuration="22.935854097s" totalImagesPullingTime="19.178012541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:50.140235856 +0000 UTC m=+23.178305780" watchObservedRunningTime="2026-07-21 13:44:50.935854097 +0000 UTC m=+23.973924017" Jul 21 13:44:51.059965 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:51.059937 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:51.734371 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:51.734267 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T13:44:50.800813095Z","UUID":"eb6b278e-ade9-411b-823f-741db26754a4","Handler":null,"Name":"","Endpoint":""} Jul 21 13:44:51.737719 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:51.737559 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 13:44:51.737719 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:51.737606 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 13:44:52.790364 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:52.790335 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:52.790722 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:52.790336 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:52.790722 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:52.790450 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:52.790722 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:52.790467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:52.790722 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:52.790579 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:52.790722 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:52.790661 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:52.925357 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:52.925277 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" event={"ID":"bf8a6a44-03aa-440b-bb95-ca157db9024c","Type":"ContainerStarted","Data":"39485908e607e476d5e7d4ab25c9bb8800b068bdd131b996ef0f10ad1ad1c554"} Jul 21 13:44:52.932523 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:52.932483 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"09fe8b5c679a3686661dbc563edcdbffd8d048073c601a9030ec3c9dfc613745"} Jul 21 13:44:52.943929 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:52.943887 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" podStartSLOduration=24.943871053 podStartE2EDuration="24.943871053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:50.954762401 +0000 UTC m=+23.992832322" watchObservedRunningTime="2026-07-21 13:44:52.943871053 +0000 UTC m=+25.981940953" Jul 21 13:44:52.944357 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:52.944334 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hs977" podStartSLOduration=2.189042671 podStartE2EDuration="23.944325927s" totalImagesPullingTime="21.755283256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:52.943995133 +0000 UTC m=+25.982065052" watchObservedRunningTime="2026-07-21 13:44:52.944325927 +0000 UTC m=+25.982395848" Jul 21 13:44:53.921357 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:53.921320 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:53.921979 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:53.921951 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:53.935646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:53.935622 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ktcmb" Jul 21 13:44:54.789808 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:54.789776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:54.789978 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:54.789778 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:54.789978 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:54.789898 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:54.789978 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:54.789903 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:54.790112 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:54.790003 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:54.790112 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:54.790093 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:55.942921 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:55.942691 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" event={"ID":"a1847b46-65d9-4412-b09c-931245edd671","Type":"ContainerStarted","Data":"08b06fba0bd7c98768471746e047b87d0bf373f191f3d22f21feb37a1ce23a5d"} Jul 21 13:44:55.943646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:55.942988 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:55.943646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:55.943038 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:55.944507 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:55.944485 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerStarted","Data":"c9598b1c5828e958955118b143e42fc2443fadacf1ebf6b2c3091d3c8fc1f12f"} Jul 21 13:44:55.960942 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:55.960915 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:55.980911 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:55.980871 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" podStartSLOduration=8.830880989 podStartE2EDuration="27.980857437s" totalImagesPullingTime="19.149976448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:44:55.979072825 +0000 UTC m=+29.017142747" watchObservedRunningTime="2026-07-21 13:44:55.980857437 +0000 UTC m=+29.018927358" Jul 21 13:44:56.790471 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:56.790297 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:56.790753 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:56.790297 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:56.790753 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:56.790576 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:56.790753 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:56.790359 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:56.790753 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:56.790658 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:56.790753 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:56.790742 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:56.947990 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:56.947959 2569 generic.go:356] "Generic (PLEG): container finished" podID="7bdb30ab-7639-4ed5-9d22-354a94d7b791" containerID="c9598b1c5828e958955118b143e42fc2443fadacf1ebf6b2c3091d3c8fc1f12f" exitCode=0 Jul 21 13:44:56.948341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:56.948045 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerDied","Data":"c9598b1c5828e958955118b143e42fc2443fadacf1ebf6b2c3091d3c8fc1f12f"} Jul 21 13:44:56.948477 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:56.948460 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:56.964697 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:56.964676 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:44:58.790407 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:58.790382 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:44:58.790857 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:58.790488 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:44:58.790857 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:58.790501 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:44:58.790857 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:58.790563 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:44:58.790857 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:58.790590 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:44:58.790857 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:44:58.790649 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:44:58.953817 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:58.953783 2569 generic.go:356] "Generic (PLEG): container finished" podID="7bdb30ab-7639-4ed5-9d22-354a94d7b791" containerID="7b8ce0c9da1e334c89cbd0b1c022cdadfcdcb88a7a3035ca878c2627d4fe8d4e" exitCode=0 Jul 21 13:44:58.954022 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:44:58.953871 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerDied","Data":"7b8ce0c9da1e334c89cbd0b1c022cdadfcdcb88a7a3035ca878c2627d4fe8d4e"} Jul 21 13:45:00.789846 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:00.789809 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:00.790229 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:00.789822 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:00.790229 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:00.789931 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:00.790229 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:00.789959 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:00.790229 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:00.790035 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:00.790229 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:00.790088 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:00.960655 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:00.960617 2569 generic.go:356] "Generic (PLEG): container finished" podID="7bdb30ab-7639-4ed5-9d22-354a94d7b791" containerID="16e6bd47e00a8ca34871bd3d61b002fe81950f5aeb617476c918fe035af771b7" exitCode=0 Jul 21 13:45:00.960808 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:00.960673 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerDied","Data":"16e6bd47e00a8ca34871bd3d61b002fe81950f5aeb617476c918fe035af771b7"} Jul 21 13:45:01.132651 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:01.132575 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:01.132816 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:01.132660 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:01.132816 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:01.132712 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 13:45:01.132816 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:01.132776 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:45:01.132816 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:01.132780 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret podName:79c9e89d-d287-46a8-b4fe-eeb2e8a31223 nodeName:}" failed. No retries permitted until 2026-07-21 13:45:33.132761378 +0000 UTC m=+66.170831278 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret") pod "global-pull-secret-syncer-ztpvd" (UID: "79c9e89d-d287-46a8-b4fe-eeb2e8a31223") : object "kube-system"/"original-pull-secret" not registered Jul 21 13:45:01.132984 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:01.132824 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs podName:68d0bf37-60ca-48b7-80f0-87d86a9858d8 nodeName:}" failed. No retries permitted until 2026-07-21 13:45:33.132812759 +0000 UTC m=+66.170882663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs") pod "network-metrics-daemon-drjp7" (UID: "68d0bf37-60ca-48b7-80f0-87d86a9858d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 13:45:01.233153 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:01.233131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:01.233271 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:01.233222 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 13:45:01.233271 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:01.233234 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 13:45:01.233271 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:01.233243 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8c9t5 for pod openshift-network-diagnostics/network-check-target-r9rll: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:45:01.233360 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:01.233282 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5 podName:c859f694-6549-4863-8b28-9e97b2b52063 nodeName:}" failed. No retries permitted until 2026-07-21 13:45:33.23327043 +0000 UTC m=+66.271340328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9t5" (UniqueName: "kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5") pod "network-check-target-r9rll" (UID: "c859f694-6549-4863-8b28-9e97b2b52063") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 13:45:02.790065 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:02.790031 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:02.790648 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:02.790031 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:02.790648 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:02.790146 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:02.790648 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:02.790031 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:02.790648 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:02.790246 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:02.790648 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:02.790329 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:04.789616 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:04.789580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:04.790070 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:04.789580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:04.790070 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:04.789720 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:04.790070 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:04.789790 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:04.790070 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:04.789594 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:04.790070 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:04.789879 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:05.974633 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:05.974561 2569 generic.go:356] "Generic (PLEG): container finished" podID="7bdb30ab-7639-4ed5-9d22-354a94d7b791" containerID="6d2de1a405f10b1077efb00b4befcf434481d01a5e99948e652ba81de627541c" exitCode=0 Jul 21 13:45:05.974633 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:05.974599 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerDied","Data":"6d2de1a405f10b1077efb00b4befcf434481d01a5e99948e652ba81de627541c"} Jul 21 13:45:06.789846 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:06.789805 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:06.790009 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:06.789805 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:06.790009 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:06.789917 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:06.790009 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:06.789950 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:06.790110 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:06.790027 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:06.790110 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:06.790079 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:06.979877 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:06.979845 2569 generic.go:356] "Generic (PLEG): container finished" podID="7bdb30ab-7639-4ed5-9d22-354a94d7b791" containerID="f7a4eb8e141851f079ccc64d7f20cdc8d078d16473fcb8284fa7cb247c77bacb" exitCode=0 Jul 21 13:45:06.980320 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:06.979903 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerDied","Data":"f7a4eb8e141851f079ccc64d7f20cdc8d078d16473fcb8284fa7cb247c77bacb"} Jul 21 13:45:07.985841 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:07.985803 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grzqj" event={"ID":"7bdb30ab-7639-4ed5-9d22-354a94d7b791","Type":"ContainerStarted","Data":"4e64253cd6223c0f964be579658b1568a14655f0f0240f03a8f83f0fc769b077"} Jul 21 13:45:08.008799 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:08.008750 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-grzqj" podStartSLOduration=8.365577862 podStartE2EDuration="40.008734232s" totalImagesPullingTime="31.64315637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:08.007649969 +0000 UTC m=+41.045719889" watchObservedRunningTime="2026-07-21 13:45:08.008734232 +0000 UTC m=+41.046804150" Jul 21 13:45:08.790387 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:08.790352 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:08.790570 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:08.790454 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:08.790570 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:08.790454 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:08.790648 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:08.790601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:08.790679 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:08.790658 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:08.790679 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:08.790594 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:10.790109 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:10.790072 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:10.790582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:10.790072 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:10.790582 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:10.790200 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:10.790582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:10.790072 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:10.790582 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:10.790283 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:10.790582 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:10.790369 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:11.641191 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:11.640994 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ztpvd"] Jul 21 13:45:11.641349 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:11.641278 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:11.641416 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:11.641374 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:11.643892 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:11.643860 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-drjp7"] Jul 21 13:45:11.644041 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:11.643972 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:11.644099 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:11.644057 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:11.654261 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:11.654234 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r9rll"] Jul 21 13:45:11.654388 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:11.654325 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:11.654448 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:11.654428 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:13.790196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:13.790170 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:13.790722 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:13.790169 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:13.790722 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:13.790275 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:13.790722 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:13.790169 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:13.790722 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:13.790342 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:13.790722 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:13.790414 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:15.790224 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:15.790194 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:15.790688 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:15.790234 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:15.790688 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:15.790276 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:15.790688 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:15.790287 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9rll" podUID="c859f694-6549-4863-8b28-9e97b2b52063" Jul 21 13:45:15.790688 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:15.790347 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drjp7" podUID="68d0bf37-60ca-48b7-80f0-87d86a9858d8" Jul 21 13:45:15.790688 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:15.790407 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztpvd" podUID="79c9e89d-d287-46a8-b4fe-eeb2e8a31223" Jul 21 13:45:16.258030 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.258003 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeReady" Jul 21 13:45:16.258182 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.258114 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 13:45:16.347022 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.346986 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w6phj"] Jul 21 13:45:16.410881 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.410855 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-82f8j"] Jul 21 13:45:16.411033 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.410985 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.414215 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.414188 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 13:45:16.414458 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.414444 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 13:45:16.414514 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.414446 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2pd27\"" Jul 21 13:45:16.430158 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.430132 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.430158 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.430150 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w6phj"] Jul 21 13:45:16.430317 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.430197 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-82f8j"] Jul 21 13:45:16.432489 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.432472 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s5m7b\"" Jul 21 13:45:16.434122 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.433993 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 13:45:16.434294 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.434270 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 13:45:16.434508 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.434487 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 13:45:16.434627 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.434511 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 13:45:16.446617 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjsk\" (UniqueName: \"kubernetes.io/projected/cd056307-610f-48de-9f6a-7c0028921262-kube-api-access-qnjsk\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.446726 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fcd41c3-839c-44d9-8085-84ce0f5c5086-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.446726 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd056307-610f-48de-9f6a-7c0028921262-metrics-tls\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.446798 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwvp\" (UniqueName: \"kubernetes.io/projected/6fcd41c3-839c-44d9-8085-84ce0f5c5086-kube-api-access-tnwvp\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.446798 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fcd41c3-839c-44d9-8085-84ce0f5c5086-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.446798 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd056307-610f-48de-9f6a-7c0028921262-tmp-dir\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.446798 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446788 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd41c3-839c-44d9-8085-84ce0f5c5086-crio-socket\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.446914 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd056307-610f-48de-9f6a-7c0028921262-config-volume\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.446914 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.446857 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fcd41c3-839c-44d9-8085-84ce0f5c5086-data-volume\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.451450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.451425 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dhvwx"] Jul 21 13:45:16.470587 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.470560 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhvwx" Jul 21 13:45:16.470744 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.470724 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhvwx"] Jul 21 13:45:16.474802 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.474773 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 13:45:16.474927 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.474861 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 13:45:16.474991 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.474927 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dghlb\"" Jul 21 13:45:16.475152 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.475138 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 13:45:16.547968 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.547883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwvp\" (UniqueName: \"kubernetes.io/projected/6fcd41c3-839c-44d9-8085-84ce0f5c5086-kube-api-access-tnwvp\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.547968 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.547921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fcd41c3-839c-44d9-8085-84ce0f5c5086-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.547968 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.547942 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd056307-610f-48de-9f6a-7c0028921262-tmp-dir\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.547968 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.547966 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd41c3-839c-44d9-8085-84ce0f5c5086-crio-socket\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.548278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.547993 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f11f3626-cec0-42a7-a05d-3bde8f5e4888-cert\") pod \"ingress-canary-dhvwx\" (UID: \"f11f3626-cec0-42a7-a05d-3bde8f5e4888\") " pod="openshift-ingress-canary/ingress-canary-dhvwx" Jul 21 13:45:16.548278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd056307-610f-48de-9f6a-7c0028921262-config-volume\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.548278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548056 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fcd41c3-839c-44d9-8085-84ce0f5c5086-data-volume\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.548278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9d47\" (UniqueName: \"kubernetes.io/projected/f11f3626-cec0-42a7-a05d-3bde8f5e4888-kube-api-access-l9d47\") pod \"ingress-canary-dhvwx\" (UID: \"f11f3626-cec0-42a7-a05d-3bde8f5e4888\") " pod="openshift-ingress-canary/ingress-canary-dhvwx" Jul 21 13:45:16.548278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjsk\" (UniqueName: \"kubernetes.io/projected/cd056307-610f-48de-9f6a-7c0028921262-kube-api-access-qnjsk\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.548278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548182 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd41c3-839c-44d9-8085-84ce0f5c5086-crio-socket\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.548278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fcd41c3-839c-44d9-8085-84ce0f5c5086-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.548278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548245 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd056307-610f-48de-9f6a-7c0028921262-metrics-tls\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.548669 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548340 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd056307-610f-48de-9f6a-7c0028921262-tmp-dir\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.548669 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548382 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fcd41c3-839c-44d9-8085-84ce0f5c5086-data-volume\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.548669 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548622 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fcd41c3-839c-44d9-8085-84ce0f5c5086-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.548806 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.548758 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd056307-610f-48de-9f6a-7c0028921262-config-volume\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.552006 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.551984 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd056307-610f-48de-9f6a-7c0028921262-metrics-tls\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.552100 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.551986 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fcd41c3-839c-44d9-8085-84ce0f5c5086-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.565583 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.565536 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjsk\" (UniqueName: \"kubernetes.io/projected/cd056307-610f-48de-9f6a-7c0028921262-kube-api-access-qnjsk\") pod \"dns-default-w6phj\" (UID: \"cd056307-610f-48de-9f6a-7c0028921262\") " pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.568598 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.568575 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwvp\" (UniqueName: \"kubernetes.io/projected/6fcd41c3-839c-44d9-8085-84ce0f5c5086-kube-api-access-tnwvp\") pod \"insights-runtime-extractor-82f8j\" (UID: \"6fcd41c3-839c-44d9-8085-84ce0f5c5086\") " pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.648759 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.648727 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f11f3626-cec0-42a7-a05d-3bde8f5e4888-cert\") pod \"ingress-canary-dhvwx\" (UID: \"f11f3626-cec0-42a7-a05d-3bde8f5e4888\") " pod="openshift-ingress-canary/ingress-canary-dhvwx" Jul 21 13:45:16.648889 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.648776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9d47\" (UniqueName: \"kubernetes.io/projected/f11f3626-cec0-42a7-a05d-3bde8f5e4888-kube-api-access-l9d47\") pod \"ingress-canary-dhvwx\" (UID: \"f11f3626-cec0-42a7-a05d-3bde8f5e4888\") " pod="openshift-ingress-canary/ingress-canary-dhvwx" Jul 21 13:45:16.650883 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.650867 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f11f3626-cec0-42a7-a05d-3bde8f5e4888-cert\") pod \"ingress-canary-dhvwx\" (UID: \"f11f3626-cec0-42a7-a05d-3bde8f5e4888\") " pod="openshift-ingress-canary/ingress-canary-dhvwx" Jul 21 13:45:16.676501 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.676474 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9d47\" (UniqueName: \"kubernetes.io/projected/f11f3626-cec0-42a7-a05d-3bde8f5e4888-kube-api-access-l9d47\") pod \"ingress-canary-dhvwx\" (UID: \"f11f3626-cec0-42a7-a05d-3bde8f5e4888\") " pod="openshift-ingress-canary/ingress-canary-dhvwx" Jul 21 13:45:16.720535 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.720510 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:16.738645 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.738624 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-82f8j" Jul 21 13:45:16.782685 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.782663 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhvwx" Jul 21 13:45:16.941384 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.941173 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w6phj"] Jul 21 13:45:16.943338 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.943317 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-82f8j"] Jul 21 13:45:16.944933 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:16.944892 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd056307_610f_48de_9f6a_7c0028921262.slice/crio-8ae6eac9359fd2ae1234309224f6984a340f669470f60178ef00809575b8b645 WatchSource:0}: Error finding container 8ae6eac9359fd2ae1234309224f6984a340f669470f60178ef00809575b8b645: Status 404 returned error can't find the container with id 8ae6eac9359fd2ae1234309224f6984a340f669470f60178ef00809575b8b645 Jul 21 13:45:16.947269 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:16.947250 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcd41c3_839c_44d9_8085_84ce0f5c5086.slice/crio-cecae94989c608ad61d52d90f22a4f20ba8d4fabb55eefd6bc9add00bcf2096d WatchSource:0}: Error finding container cecae94989c608ad61d52d90f22a4f20ba8d4fabb55eefd6bc9add00bcf2096d: Status 404 returned error can't find the container with id cecae94989c608ad61d52d90f22a4f20ba8d4fabb55eefd6bc9add00bcf2096d Jul 21 13:45:16.951055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:16.951031 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhvwx"] Jul 21 13:45:16.954288 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:16.954265 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11f3626_cec0_42a7_a05d_3bde8f5e4888.slice/crio-cd6b8b1350803f2b38dde6046ed63f2bc60b64684d008b40c410547fdc604e52 WatchSource:0}: Error finding container cd6b8b1350803f2b38dde6046ed63f2bc60b64684d008b40c410547fdc604e52: Status 404 returned error can't find the container with id cd6b8b1350803f2b38dde6046ed63f2bc60b64684d008b40c410547fdc604e52 Jul 21 13:45:17.006202 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.006171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhvwx" event={"ID":"f11f3626-cec0-42a7-a05d-3bde8f5e4888","Type":"ContainerStarted","Data":"cd6b8b1350803f2b38dde6046ed63f2bc60b64684d008b40c410547fdc604e52"} Jul 21 13:45:17.007189 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.007164 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82f8j" event={"ID":"6fcd41c3-839c-44d9-8085-84ce0f5c5086","Type":"ContainerStarted","Data":"cecae94989c608ad61d52d90f22a4f20ba8d4fabb55eefd6bc9add00bcf2096d"} Jul 21 13:45:17.008179 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.008159 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6phj" event={"ID":"cd056307-610f-48de-9f6a-7c0028921262","Type":"ContainerStarted","Data":"8ae6eac9359fd2ae1234309224f6984a340f669470f60178ef00809575b8b645"} Jul 21 13:45:17.145750 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.145644 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-czwsf"] Jul 21 13:45:17.154222 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.154200 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-czwsf" Jul 21 13:45:17.157219 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.157197 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zgm6v\"" Jul 21 13:45:17.157362 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.157343 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 13:45:17.157583 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.157569 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 13:45:17.167269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.167241 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-czwsf"] Jul 21 13:45:17.235669 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.235637 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj"] Jul 21 13:45:17.247619 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.247590 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" Jul 21 13:45:17.248587 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.248557 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj"] Jul 21 13:45:17.250305 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.250284 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 13:45:17.250687 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.250672 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xxjcr\"" Jul 21 13:45:17.251588 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.251571 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5jb\" (UniqueName: \"kubernetes.io/projected/2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde-kube-api-access-ql5jb\") pod \"downloads-5bc67dcd46-czwsf\" (UID: \"2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde\") " pod="openshift-console/downloads-5bc67dcd46-czwsf" Jul 21 13:45:17.352115 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.352085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c20e3db2-ed3c-4118-9171-90aae7440577-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-frjlj\" (UID: \"c20e3db2-ed3c-4118-9171-90aae7440577\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" Jul 21 13:45:17.352267 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.352132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5jb\" (UniqueName: \"kubernetes.io/projected/2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde-kube-api-access-ql5jb\") pod \"downloads-5bc67dcd46-czwsf\" (UID: \"2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde\") " pod="openshift-console/downloads-5bc67dcd46-czwsf" Jul 21 13:45:17.363416 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.363390 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5jb\" (UniqueName: \"kubernetes.io/projected/2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde-kube-api-access-ql5jb\") pod \"downloads-5bc67dcd46-czwsf\" (UID: \"2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde\") " pod="openshift-console/downloads-5bc67dcd46-czwsf" Jul 21 13:45:17.453215 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.453130 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c20e3db2-ed3c-4118-9171-90aae7440577-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-frjlj\" (UID: \"c20e3db2-ed3c-4118-9171-90aae7440577\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" Jul 21 13:45:17.455923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.455899 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c20e3db2-ed3c-4118-9171-90aae7440577-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-frjlj\" (UID: \"c20e3db2-ed3c-4118-9171-90aae7440577\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" Jul 21 13:45:17.464081 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.464046 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-czwsf" Jul 21 13:45:17.556524 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.556493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" Jul 21 13:45:17.622415 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.622383 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-czwsf"] Jul 21 13:45:17.632849 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:17.632819 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0ef78a_f8ca_4c56_88b5_4c8e54e09fde.slice/crio-fa2ebc520d6e9cc9f72f1439bc2aa8b470a9c9694453a130a2d568accd3745d4 WatchSource:0}: Error finding container fa2ebc520d6e9cc9f72f1439bc2aa8b470a9c9694453a130a2d568accd3745d4: Status 404 returned error can't find the container with id fa2ebc520d6e9cc9f72f1439bc2aa8b470a9c9694453a130a2d568accd3745d4 Jul 21 13:45:17.718147 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.718070 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj"] Jul 21 13:45:17.722604 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:17.722570 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20e3db2_ed3c_4118_9171_90aae7440577.slice/crio-0f0b8bc0e4662eb81f502d7bfcbb766d52d80cf0d774daaa84d66fa48a208313 WatchSource:0}: Error finding container 0f0b8bc0e4662eb81f502d7bfcbb766d52d80cf0d774daaa84d66fa48a208313: Status 404 returned error can't find the container with id 0f0b8bc0e4662eb81f502d7bfcbb766d52d80cf0d774daaa84d66fa48a208313 Jul 21 13:45:17.790062 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.790002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:17.790285 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.790015 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:17.793067 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.790009 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:17.794063 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.793637 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 13:45:17.794063 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.793862 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 13:45:17.794063 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.793896 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-29gkd\"" Jul 21 13:45:17.794063 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.793938 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pt8sb\"" Jul 21 13:45:17.794063 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.794065 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 13:45:17.794364 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:17.794187 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 13:45:18.013294 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:18.013253 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" event={"ID":"c20e3db2-ed3c-4118-9171-90aae7440577","Type":"ContainerStarted","Data":"0f0b8bc0e4662eb81f502d7bfcbb766d52d80cf0d774daaa84d66fa48a208313"} Jul 21 13:45:18.015331 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:18.015295 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-czwsf" event={"ID":"2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde","Type":"ContainerStarted","Data":"fa2ebc520d6e9cc9f72f1439bc2aa8b470a9c9694453a130a2d568accd3745d4"} Jul 21 13:45:25.040212 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.040062 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhvwx" event={"ID":"f11f3626-cec0-42a7-a05d-3bde8f5e4888","Type":"ContainerStarted","Data":"f49e210c9c2a1945f8af348795f9aa43ca9a6abd642acbe694f5a2219f2babfd"} Jul 21 13:45:25.041614 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.041577 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82f8j" event={"ID":"6fcd41c3-839c-44d9-8085-84ce0f5c5086","Type":"ContainerStarted","Data":"2abc848286ab3932a7b3d55412c70243e6532638538e06883d0237063fef2ac6"} Jul 21 13:45:25.043133 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.043106 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" event={"ID":"c20e3db2-ed3c-4118-9171-90aae7440577","Type":"ContainerStarted","Data":"bbc09c6c97200c0a7cffbe00624b3fdfaa9718ea9b294e24e645a01601ed8da5"} Jul 21 13:45:25.043297 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.043276 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" Jul 21 13:45:25.045198 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.045064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6phj" event={"ID":"cd056307-610f-48de-9f6a-7c0028921262","Type":"ContainerStarted","Data":"b81276eba4bc58e0fff4f249b608b3fbda7e491409661bb7745082a97e4634be"} Jul 21 13:45:25.045198 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.045107 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6phj" event={"ID":"cd056307-610f-48de-9f6a-7c0028921262","Type":"ContainerStarted","Data":"be244d5b7004123a36743b25ccb3bc7f9a8433ee6bebfa80dfebd0c0d7d9dc4b"} Jul 21 13:45:25.045198 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.045157 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:25.048887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.048853 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" Jul 21 13:45:25.059301 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.059266 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dhvwx" podStartSLOduration=1.479832128 podStartE2EDuration="9.059249109s" totalImagesPullingTime="7.579416981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:25.058647603 +0000 UTC m=+58.096717523" watchObservedRunningTime="2026-07-21 13:45:25.059249109 +0000 UTC m=+58.097319035" Jul 21 13:45:25.078610 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.078556 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-frjlj" podStartSLOduration=1.659116542 podStartE2EDuration="8.078520744s" totalImagesPullingTime="6.419404202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:25.078159185 +0000 UTC m=+58.116229107" watchObservedRunningTime="2026-07-21 13:45:25.078520744 +0000 UTC m=+58.116590669" Jul 21 13:45:25.097324 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.097274 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w6phj" podStartSLOduration=1.508308967 podStartE2EDuration="9.097253831s" totalImagesPullingTime="7.588944864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:25.095911886 +0000 UTC m=+58.133981810" watchObservedRunningTime="2026-07-21 13:45:25.097253831 +0000 UTC m=+58.135323753" Jul 21 13:45:25.246477 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.246443 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-ngt84"] Jul 21 13:45:25.249565 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.249530 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.253444 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.253388 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 13:45:25.253444 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.253407 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 13:45:25.253641 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.253465 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-64gm7\"" Jul 21 13:45:25.253641 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.253489 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 13:45:25.253641 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.253465 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 13:45:25.253641 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.253617 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 13:45:25.263597 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.263576 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-ngt84"] Jul 21 13:45:25.316839 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.316763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.316839 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.316809 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26m7j\" (UniqueName: \"kubernetes.io/projected/b91a43c7-5d49-493c-b568-1e96dca4340d-kube-api-access-26m7j\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.317030 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.316874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b91a43c7-5d49-493c-b568-1e96dca4340d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.317030 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.316944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.418043 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.417474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.418043 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.417521 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26m7j\" (UniqueName: \"kubernetes.io/projected/b91a43c7-5d49-493c-b568-1e96dca4340d-kube-api-access-26m7j\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.418043 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.417569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b91a43c7-5d49-493c-b568-1e96dca4340d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.418043 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.417627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.418390 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:25.418118 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 21 13:45:25.418390 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:25.418189 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-tls podName:b91a43c7-5d49-493c-b568-1e96dca4340d nodeName:}" failed. No retries permitted until 2026-07-21 13:45:25.91816928 +0000 UTC m=+58.956239198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-ngt84" (UID: "b91a43c7-5d49-493c-b568-1e96dca4340d") : secret "prometheus-operator-tls" not found Jul 21 13:45:25.418724 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.418698 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b91a43c7-5d49-493c-b568-1e96dca4340d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.422026 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.421962 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.433080 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.433020 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26m7j\" (UniqueName: \"kubernetes.io/projected/b91a43c7-5d49-493c-b568-1e96dca4340d-kube-api-access-26m7j\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.922773 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.922730 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:25.926603 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:25.926530 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b91a43c7-5d49-493c-b568-1e96dca4340d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-ngt84\" (UID: \"b91a43c7-5d49-493c-b568-1e96dca4340d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:26.158815 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.158778 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" Jul 21 13:45:26.308153 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.308120 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-ngt84"] Jul 21 13:45:26.311856 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:26.311825 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91a43c7_5d49_493c_b568_1e96dca4340d.slice/crio-0497e2a47abdaf8fe77b9ec3afec1756efc3fd5ed94f3144c02642171077a18e WatchSource:0}: Error finding container 0497e2a47abdaf8fe77b9ec3afec1756efc3fd5ed94f3144c02642171077a18e: Status 404 returned error can't find the container with id 0497e2a47abdaf8fe77b9ec3afec1756efc3fd5ed94f3144c02642171077a18e Jul 21 13:45:26.443561 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.443358 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-768987b95f-fh2b8"] Jul 21 13:45:26.447636 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.447610 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.450280 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.450258 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 13:45:26.450392 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.450369 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-68n6s\"" Jul 21 13:45:26.450616 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.450600 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 13:45:26.450688 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.450629 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 13:45:26.451286 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.451261 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 13:45:26.451384 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.451271 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 13:45:26.460716 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.460662 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768987b95f-fh2b8"] Jul 21 13:45:26.528391 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.528354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-oauth-serving-cert\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.528571 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.528401 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-serving-cert\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.528571 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.528432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvtq\" (UniqueName: \"kubernetes.io/projected/58be5d89-df08-45b5-8fed-5afc5a84f585-kube-api-access-7qvtq\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.528700 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.528561 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-oauth-config\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.528700 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.528595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-console-config\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.528700 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.528622 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-service-ca\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.630415 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.630012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-oauth-config\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.630415 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.630062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-console-config\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.630415 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.630093 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-service-ca\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.630415 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.630177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-oauth-serving-cert\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.630415 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.630203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-serving-cert\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.630415 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.630231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvtq\" (UniqueName: \"kubernetes.io/projected/58be5d89-df08-45b5-8fed-5afc5a84f585-kube-api-access-7qvtq\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.630878 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.630838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-console-config\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.630993 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.630967 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-service-ca\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.631102 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.631064 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-oauth-serving-cert\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.632876 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.632853 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-oauth-config\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.632991 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.632951 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-serving-cert\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.644445 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.644424 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvtq\" (UniqueName: \"kubernetes.io/projected/58be5d89-df08-45b5-8fed-5afc5a84f585-kube-api-access-7qvtq\") pod \"console-768987b95f-fh2b8\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:26.760274 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:26.760241 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:27.051583 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:27.051528 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" event={"ID":"b91a43c7-5d49-493c-b568-1e96dca4340d","Type":"ContainerStarted","Data":"0497e2a47abdaf8fe77b9ec3afec1756efc3fd5ed94f3144c02642171077a18e"} Jul 21 13:45:28.966435 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:28.966406 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfr5" Jul 21 13:45:33.186532 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.186486 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:33.187030 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.186590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:33.189660 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.189631 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 13:45:33.190230 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.190211 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 13:45:33.199414 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.199393 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79c9e89d-d287-46a8-b4fe-eeb2e8a31223-original-pull-secret\") pod \"global-pull-secret-syncer-ztpvd\" (UID: \"79c9e89d-d287-46a8-b4fe-eeb2e8a31223\") " pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:33.199564 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.199487 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0bf37-60ca-48b7-80f0-87d86a9858d8-metrics-certs\") pod \"network-metrics-daemon-drjp7\" (UID: \"68d0bf37-60ca-48b7-80f0-87d86a9858d8\") " pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:33.287066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.287021 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:33.290021 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.289997 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 13:45:33.299682 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.299660 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 13:45:33.310378 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.310353 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9t5\" (UniqueName: \"kubernetes.io/projected/c859f694-6549-4863-8b28-9e97b2b52063-kube-api-access-8c9t5\") pod \"network-check-target-r9rll\" (UID: \"c859f694-6549-4863-8b28-9e97b2b52063\") " pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:33.409351 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.409321 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-29gkd\"" Jul 21 13:45:33.415636 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.415615 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztpvd" Jul 21 13:45:33.416687 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.416666 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:33.426895 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.426873 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pt8sb\"" Jul 21 13:45:33.435194 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:33.435168 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drjp7" Jul 21 13:45:35.022141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.022097 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c8f77dd4d-5hzf4"] Jul 21 13:45:35.044176 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.044149 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8f77dd4d-5hzf4"] Jul 21 13:45:35.044369 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.044268 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.050487 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.050457 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w6phj" Jul 21 13:45:35.057168 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.057137 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 13:45:35.096672 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.096637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-oauth-serving-cert\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.096860 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.096679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-serving-cert\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.096860 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.096728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-trusted-ca-bundle\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.096860 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.096747 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-oauth-config\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.096860 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.096791 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-console-config\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.097062 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.096875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz2kg\" (UniqueName: \"kubernetes.io/projected/4790404b-c94b-40f6-9ffc-932b6798c742-kube-api-access-wz2kg\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.097062 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.096910 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-service-ca\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.197871 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.197829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2kg\" (UniqueName: \"kubernetes.io/projected/4790404b-c94b-40f6-9ffc-932b6798c742-kube-api-access-wz2kg\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.198054 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.197885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-service-ca\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.198054 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.197911 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-oauth-serving-cert\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.198054 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.197952 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-serving-cert\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.198054 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.197985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-trusted-ca-bundle\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.198054 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.198010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-oauth-config\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.198054 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.198042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-console-config\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.198883 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.198854 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-console-config\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.199007 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.198854 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-service-ca\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.199007 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.198853 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-oauth-serving-cert\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.199121 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.199050 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-trusted-ca-bundle\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.200981 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.200954 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-oauth-config\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.201387 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.201355 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-serving-cert\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.205974 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.205951 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2kg\" (UniqueName: \"kubernetes.io/projected/4790404b-c94b-40f6-9ffc-932b6798c742-kube-api-access-wz2kg\") pod \"console-c8f77dd4d-5hzf4\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:35.359527 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:35.359441 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:36.308513 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:36.308480 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768987b95f-fh2b8"] Jul 21 13:45:36.372237 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:36.372206 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58be5d89_df08_45b5_8fed_5afc5a84f585.slice/crio-ffce84705fe6bf5559fda42cea54347a851f1d82fffefd6e30143bb2b56d655e WatchSource:0}: Error finding container ffce84705fe6bf5559fda42cea54347a851f1d82fffefd6e30143bb2b56d655e: Status 404 returned error can't find the container with id ffce84705fe6bf5559fda42cea54347a851f1d82fffefd6e30143bb2b56d655e Jul 21 13:45:36.903402 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:36.903319 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r9rll"] Jul 21 13:45:36.915940 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:36.915900 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8f77dd4d-5hzf4"] Jul 21 13:45:36.924153 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:36.923470 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4790404b_c94b_40f6_9ffc_932b6798c742.slice/crio-6d93ec6ce96f6a08670011a17d4ab4dca5c3c45c2d33996e9315d0068a0a7ce7 WatchSource:0}: Error finding container 6d93ec6ce96f6a08670011a17d4ab4dca5c3c45c2d33996e9315d0068a0a7ce7: Status 404 returned error can't find the container with id 6d93ec6ce96f6a08670011a17d4ab4dca5c3c45c2d33996e9315d0068a0a7ce7 Jul 21 13:45:36.953100 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:36.953027 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-drjp7"] Jul 21 13:45:36.956732 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:36.956696 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d0bf37_60ca_48b7_80f0_87d86a9858d8.slice/crio-c0b72235da0255f3de4d1ece76732a3c093773c90e386227f5f2a45a85b95964 WatchSource:0}: Error finding container c0b72235da0255f3de4d1ece76732a3c093773c90e386227f5f2a45a85b95964: Status 404 returned error can't find the container with id c0b72235da0255f3de4d1ece76732a3c093773c90e386227f5f2a45a85b95964 Jul 21 13:45:36.966961 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:36.966938 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ztpvd"] Jul 21 13:45:36.969981 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:36.969946 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c9e89d_d287_46a8_b4fe_eeb2e8a31223.slice/crio-05a52e4e0e35c3ff9a2dd07270775fac47d943ed5014c269bb10312fed565484 WatchSource:0}: Error finding container 05a52e4e0e35c3ff9a2dd07270775fac47d943ed5014c269bb10312fed565484: Status 404 returned error can't find the container with id 05a52e4e0e35c3ff9a2dd07270775fac47d943ed5014c269bb10312fed565484 Jul 21 13:45:37.083850 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.083762 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-czwsf" event={"ID":"2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde","Type":"ContainerStarted","Data":"dc5ad7e9905667fa155cbac9920d90151b1da8a2b5144aa463d560e9ca62971a"} Jul 21 13:45:37.084215 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.084161 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-czwsf" Jul 21 13:45:37.086269 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.086227 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" event={"ID":"b91a43c7-5d49-493c-b568-1e96dca4340d","Type":"ContainerStarted","Data":"32e5d32f01fb8bf0da2da981cf04b392efe6b4b12afc98b1b141e21b4fda0dab"} Jul 21 13:45:37.086408 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.086283 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" event={"ID":"b91a43c7-5d49-493c-b568-1e96dca4340d","Type":"ContainerStarted","Data":"beecc92724e552f7566038a31b43a9e88827ae53e1845bc68e3a7b6db5835af4"} Jul 21 13:45:37.087659 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.087633 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ztpvd" event={"ID":"79c9e89d-d287-46a8-b4fe-eeb2e8a31223","Type":"ContainerStarted","Data":"05a52e4e0e35c3ff9a2dd07270775fac47d943ed5014c269bb10312fed565484"} Jul 21 13:45:37.088895 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.088872 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-drjp7" event={"ID":"68d0bf37-60ca-48b7-80f0-87d86a9858d8","Type":"ContainerStarted","Data":"c0b72235da0255f3de4d1ece76732a3c093773c90e386227f5f2a45a85b95964"} Jul 21 13:45:37.089920 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.089898 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8f77dd4d-5hzf4" event={"ID":"4790404b-c94b-40f6-9ffc-932b6798c742","Type":"ContainerStarted","Data":"6d93ec6ce96f6a08670011a17d4ab4dca5c3c45c2d33996e9315d0068a0a7ce7"} Jul 21 13:45:37.090955 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.090934 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r9rll" event={"ID":"c859f694-6549-4863-8b28-9e97b2b52063","Type":"ContainerStarted","Data":"ca22d92c78e8ce1dbaad7e4ca9b605ca1120b97aefb59d92245c3dea76412ebb"} Jul 21 13:45:37.092026 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.092006 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768987b95f-fh2b8" event={"ID":"58be5d89-df08-45b5-8fed-5afc5a84f585","Type":"ContainerStarted","Data":"ffce84705fe6bf5559fda42cea54347a851f1d82fffefd6e30143bb2b56d655e"} Jul 21 13:45:37.093705 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.093684 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82f8j" event={"ID":"6fcd41c3-839c-44d9-8085-84ce0f5c5086","Type":"ContainerStarted","Data":"a6e717a7b5ee5bb35c890e5a3aeb754ebbbd24d1b58dc1f80cfd4dec3d629d72"} Jul 21 13:45:37.096512 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.096493 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-czwsf" Jul 21 13:45:37.103464 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.103415 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-czwsf" podStartSLOduration=1.030891778 podStartE2EDuration="20.103400906s" totalImagesPullingTime="19.072509128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:37.102875261 +0000 UTC m=+70.140945181" watchObservedRunningTime="2026-07-21 13:45:37.103400906 +0000 UTC m=+70.141470828" Jul 21 13:45:37.140602 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:37.140439 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ngt84" podStartSLOduration=1.715083833 podStartE2EDuration="12.140420556s" totalImagesPullingTime="10.425336723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:37.139630577 +0000 UTC m=+70.177700522" watchObservedRunningTime="2026-07-21 13:45:37.140420556 +0000 UTC m=+70.178490478" Jul 21 13:45:38.649190 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.649152 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn"] Jul 21 13:45:38.678977 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.678939 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.682858 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.682287 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 21 13:45:38.682858 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.682690 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 21 13:45:38.684580 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.684464 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn"] Jul 21 13:45:38.687943 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.687646 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-226rs\"" Jul 21 13:45:38.708930 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.707621 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xgrdw"] Jul 21 13:45:38.727732 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.727705 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-g896z"] Jul 21 13:45:38.730989 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.728007 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.731405 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.731298 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.731405 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.731338 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba604b91-8658-4512-93f3-724667407ea1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.731661 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.731634 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.731789 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.731682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rcl\" (UniqueName: \"kubernetes.io/projected/ba604b91-8658-4512-93f3-724667407ea1-kube-api-access-59rcl\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.733165 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.733141 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 13:45:38.741789 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.739054 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t6whq\"" Jul 21 13:45:38.741789 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.739282 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 13:45:38.741789 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.739559 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 13:45:38.749853 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.749830 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-g896z"] Jul 21 13:45:38.749965 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.749944 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.754735 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.754712 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 21 13:45:38.754973 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.754954 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 21 13:45:38.755580 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.755561 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-flqnn\"" Jul 21 13:45:38.755785 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.755767 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 21 13:45:38.833008 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.832962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9063bc3f-f3cb-4946-9841-b47b6503b7b4-metrics-client-ca\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833203 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-sys\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833203 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.833203 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833093 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.833203 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba604b91-8658-4512-93f3-724667407ea1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.833203 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.833203 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833179 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-wtmp\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833214 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1e67cdf6-b024-4463-879e-b95846974683-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833242 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833273 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsd6c\" (UniqueName: \"kubernetes.io/projected/9063bc3f-f3cb-4946-9841-b47b6503b7b4-kube-api-access-wsd6c\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833303 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833328 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdw8m\" (UniqueName: \"kubernetes.io/projected/1e67cdf6-b024-4463-879e-b95846974683-kube-api-access-qdw8m\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833380 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-root\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833404 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833438 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rcl\" (UniqueName: \"kubernetes.io/projected/ba604b91-8658-4512-93f3-724667407ea1-kube-api-access-59rcl\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-textfile\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-tls\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.833526 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.833526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e67cdf6-b024-4463-879e-b95846974683-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.834147 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:38.833722 2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 21 13:45:38.834147 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:38.833798 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-tls podName:ba604b91-8658-4512-93f3-724667407ea1 nodeName:}" failed. No retries permitted until 2026-07-21 13:45:39.33377672 +0000 UTC m=+72.371846649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-dtcsn" (UID: "ba604b91-8658-4512-93f3-724667407ea1") : secret "openshift-state-metrics-tls" not found Jul 21 13:45:38.834848 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.834822 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba604b91-8658-4512-93f3-724667407ea1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.841766 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.841704 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.878816 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.878497 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rcl\" (UniqueName: \"kubernetes.io/projected/ba604b91-8658-4512-93f3-724667407ea1-kube-api-access-59rcl\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:38.934721 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.934394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.934721 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.934442 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdw8m\" (UniqueName: \"kubernetes.io/projected/1e67cdf6-b024-4463-879e-b95846974683-kube-api-access-qdw8m\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.934721 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.934471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-root\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.934721 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.934500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.934721 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.934532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-textfile\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.934721 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.934589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-tls\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.934721 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.934618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e67cdf6-b024-4463-879e-b95846974683-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.935277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9063bc3f-f3cb-4946-9841-b47b6503b7b4-metrics-client-ca\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.935321 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-sys\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.935348 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.935406 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.935433 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-wtmp\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.935465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1e67cdf6-b024-4463-879e-b95846974683-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.935490 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.935531 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsd6c\" (UniqueName: \"kubernetes.io/projected/9063bc3f-f3cb-4946-9841-b47b6503b7b4-kube-api-access-wsd6c\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.936454 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.937183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.936755 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-root\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.939696 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.937940 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-wtmp\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.939696 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.938520 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9063bc3f-f3cb-4946-9841-b47b6503b7b4-sys\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.939696 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.938921 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1e67cdf6-b024-4463-879e-b95846974683-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.939696 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:38.939087 2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 21 13:45:38.939696 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:45:38.939183 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-tls podName:1e67cdf6-b024-4463-879e-b95846974683 nodeName:}" failed. No retries permitted until 2026-07-21 13:45:39.439162224 +0000 UTC m=+72.477232129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-g896z" (UID: "1e67cdf6-b024-4463-879e-b95846974683") : secret "kube-state-metrics-tls" not found Jul 21 13:45:38.939696 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.939591 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9063bc3f-f3cb-4946-9841-b47b6503b7b4-metrics-client-ca\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.939696 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.939623 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-textfile\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.939696 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.939623 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.940565 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.940522 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e67cdf6-b024-4463-879e-b95846974683-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.944572 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.944076 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:38.946669 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.946619 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-tls\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.947626 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.947574 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9063bc3f-f3cb-4946-9841-b47b6503b7b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.949317 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.949269 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsd6c\" (UniqueName: \"kubernetes.io/projected/9063bc3f-f3cb-4946-9841-b47b6503b7b4-kube-api-access-wsd6c\") pod \"node-exporter-xgrdw\" (UID: \"9063bc3f-f3cb-4946-9841-b47b6503b7b4\") " pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:38.951812 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:38.951769 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdw8m\" (UniqueName: \"kubernetes.io/projected/1e67cdf6-b024-4463-879e-b95846974683-kube-api-access-qdw8m\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:39.046382 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.046122 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xgrdw" Jul 21 13:45:39.092822 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:39.092776 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9063bc3f_f3cb_4946_9841_b47b6503b7b4.slice/crio-282023222f0213089269f03c44676b9e8c126cb04e7a28b49f020e00fb45e4c1 WatchSource:0}: Error finding container 282023222f0213089269f03c44676b9e8c126cb04e7a28b49f020e00fb45e4c1: Status 404 returned error can't find the container with id 282023222f0213089269f03c44676b9e8c126cb04e7a28b49f020e00fb45e4c1 Jul 21 13:45:39.124504 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.124421 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrdw" event={"ID":"9063bc3f-f3cb-4946-9841-b47b6503b7b4","Type":"ContainerStarted","Data":"282023222f0213089269f03c44676b9e8c126cb04e7a28b49f020e00fb45e4c1"} Jul 21 13:45:39.341321 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.341056 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:39.347093 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.347035 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba604b91-8658-4512-93f3-724667407ea1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-dtcsn\" (UID: \"ba604b91-8658-4512-93f3-724667407ea1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:39.442616 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.442517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:39.456325 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.456267 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e67cdf6-b024-4463-879e-b95846974683-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-g896z\" (UID: \"1e67cdf6-b024-4463-879e-b95846974683\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:39.599972 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.599703 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" Jul 21 13:45:39.668479 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.668249 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" Jul 21 13:45:39.729568 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.728025 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:45:39.752618 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.752407 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:45:39.752618 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.752605 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.765731 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.763111 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 21 13:45:39.765731 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.763339 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 21 13:45:39.776495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.775867 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dskmg\"" Jul 21 13:45:39.776495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.776106 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 21 13:45:39.776495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.776240 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 21 13:45:39.776495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.776360 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 21 13:45:39.778838 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.777612 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 21 13:45:39.778838 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.778124 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 21 13:45:39.778838 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.778309 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 21 13:45:39.778838 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.778381 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 21 13:45:39.853801 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853593 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-web-config\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.853801 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853647 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.853801 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853681 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.853801 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.853801 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-config-volume\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.853801 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.854192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.854192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.854192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853901 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-config-out\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.854192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853927 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.854192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.854192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.853971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k288f\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-kube-api-access-k288f\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.854192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.854002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955608 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-config-out\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955685 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k288f\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-kube-api-access-k288f\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955778 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-web-config\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955879 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-config-volume\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.955949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.956702 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.955977 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.956702 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.956001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.956702 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.956374 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.959612 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.956932 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.959612 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.959266 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.980814 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.980761 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.982440 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.982309 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k288f\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-kube-api-access-k288f\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.985138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.985067 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-web-config\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.985921 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.985876 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.986027 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.986000 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.986486 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.986443 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-config-out\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.987690 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.987664 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.988186 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.988161 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.988959 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.988920 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:39.991039 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:39.990996 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-config-volume\") pod \"alertmanager-main-0\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:40.081570 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.080680 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:45:40.663359 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.651914 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5656dbdcdc-njbbd"] Jul 21 13:45:40.698842 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.690513 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5656dbdcdc-njbbd"] Jul 21 13:45:40.698842 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.690723 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.698842 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.696167 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bajlnua8i7a3a\"" Jul 21 13:45:40.698842 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.696268 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 13:45:40.698842 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.696432 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5lwcl\"" Jul 21 13:45:40.698842 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.696606 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 13:45:40.700025 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.699309 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 13:45:40.700025 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.699514 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 13:45:40.700025 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.699753 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 13:45:40.727273 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.727245 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-g896z"] Jul 21 13:45:40.770304 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.770224 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.770627 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.770589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20c56c13-3380-4a02-9d22-7f85f7ec346c-metrics-client-ca\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.770752 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.770643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-tls\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.770752 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.770691 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcpr\" (UniqueName: \"kubernetes.io/projected/20c56c13-3380-4a02-9d22-7f85f7ec346c-kube-api-access-2pcpr\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.770854 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.770720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.770854 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.770798 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-grpc-tls\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.770960 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.770912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.771039 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.771016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.811906 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.811878 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:45:40.871676 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.871628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.871853 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.871691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.871853 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.871750 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.871853 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.871799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20c56c13-3380-4a02-9d22-7f85f7ec346c-metrics-client-ca\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.871853 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.871831 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-tls\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.872061 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.871857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcpr\" (UniqueName: \"kubernetes.io/projected/20c56c13-3380-4a02-9d22-7f85f7ec346c-kube-api-access-2pcpr\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.872061 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.871883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.872061 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.871920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-grpc-tls\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.873792 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.873075 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20c56c13-3380-4a02-9d22-7f85f7ec346c-metrics-client-ca\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.875768 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.875621 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-tls\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.877861 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.877834 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.878527 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.878486 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-grpc-tls\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.878653 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.878568 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.888604 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.888520 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.888719 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.888690 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20c56c13-3380-4a02-9d22-7f85f7ec346c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.890906 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.890866 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcpr\" (UniqueName: \"kubernetes.io/projected/20c56c13-3380-4a02-9d22-7f85f7ec346c-kube-api-access-2pcpr\") pod \"thanos-querier-5656dbdcdc-njbbd\" (UID: \"20c56c13-3380-4a02-9d22-7f85f7ec346c\") " pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:40.915520 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:40.915355 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e67cdf6_b024_4463_879e_b95846974683.slice/crio-828e1e32c478289b19a9e50ed322050c232df17d7eeee7c2ccd2dabfb001ae18 WatchSource:0}: Error finding container 828e1e32c478289b19a9e50ed322050c232df17d7eeee7c2ccd2dabfb001ae18: Status 404 returned error can't find the container with id 828e1e32c478289b19a9e50ed322050c232df17d7eeee7c2ccd2dabfb001ae18 Jul 21 13:45:40.916277 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:40.916240 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cfef40_9134_4b59_887c_097dadf6a797.slice/crio-8273bf1e41d681eb8a46e8c48f33828a85b4618a5ba99f7fcc5d90911d6cd94f WatchSource:0}: Error finding container 8273bf1e41d681eb8a46e8c48f33828a85b4618a5ba99f7fcc5d90911d6cd94f: Status 404 returned error can't find the container with id 8273bf1e41d681eb8a46e8c48f33828a85b4618a5ba99f7fcc5d90911d6cd94f Jul 21 13:45:40.955406 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:40.955288 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn"] Jul 21 13:45:40.962716 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:40.962683 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba604b91_8658_4512_93f3_724667407ea1.slice/crio-884777d38eb99d94d115801ef96173b899a1350abe555e66db11bb0c2a624411 WatchSource:0}: Error finding container 884777d38eb99d94d115801ef96173b899a1350abe555e66db11bb0c2a624411: Status 404 returned error can't find the container with id 884777d38eb99d94d115801ef96173b899a1350abe555e66db11bb0c2a624411 Jul 21 13:45:41.012152 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:41.011938 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:45:41.160307 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:41.157815 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" event={"ID":"ba604b91-8658-4512-93f3-724667407ea1","Type":"ContainerStarted","Data":"cb1f0f03817864fe8d76dd04a2a77a48b997e844d092cd7d7c572d53474f2d53"} Jul 21 13:45:41.160307 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:41.157886 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" event={"ID":"ba604b91-8658-4512-93f3-724667407ea1","Type":"ContainerStarted","Data":"884777d38eb99d94d115801ef96173b899a1350abe555e66db11bb0c2a624411"} Jul 21 13:45:41.160307 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:41.159703 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" event={"ID":"1e67cdf6-b024-4463-879e-b95846974683","Type":"ContainerStarted","Data":"828e1e32c478289b19a9e50ed322050c232df17d7eeee7c2ccd2dabfb001ae18"} Jul 21 13:45:41.162229 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:41.162126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerStarted","Data":"8273bf1e41d681eb8a46e8c48f33828a85b4618a5ba99f7fcc5d90911d6cd94f"} Jul 21 13:45:41.285777 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:41.285281 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5656dbdcdc-njbbd"] Jul 21 13:45:42.177148 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.176180 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" event={"ID":"20c56c13-3380-4a02-9d22-7f85f7ec346c","Type":"ContainerStarted","Data":"7035e96c71bda53ff1092e6501d1ab5e26514e9bbadf50a7fe5ca5d37d7dce87"} Jul 21 13:45:42.181912 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.181311 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-drjp7" event={"ID":"68d0bf37-60ca-48b7-80f0-87d86a9858d8","Type":"ContainerStarted","Data":"841572ef86c4dfdfc943ae5c1ee0242ff2bbb9a68426eab77276e643368af791"} Jul 21 13:45:42.181912 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.181362 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-drjp7" event={"ID":"68d0bf37-60ca-48b7-80f0-87d86a9858d8","Type":"ContainerStarted","Data":"2c796f724d1ef11b61cd66a309fb7999fb527606d528869c5a0493555c0b8ec8"} Jul 21 13:45:42.190511 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.190478 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" event={"ID":"ba604b91-8658-4512-93f3-724667407ea1","Type":"ContainerStarted","Data":"6ca1e3f1753fcbe830b196916f26dbd1bdda1ec7d7282cf4ee090fbf66ac0363"} Jul 21 13:45:42.198805 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.198745 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-drjp7" podStartSLOduration=70.209050121 podStartE2EDuration="1m14.198727996s" totalImagesPullingTime="3.989677875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:42.196677797 +0000 UTC m=+75.234747741" watchObservedRunningTime="2026-07-21 13:45:42.198727996 +0000 UTC m=+75.236797908" Jul 21 13:45:42.969688 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.968943 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b857c966-2pwpf"] Jul 21 13:45:42.994347 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.994296 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b857c966-2pwpf"] Jul 21 13:45:42.994518 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.994434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:42.998983 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.997607 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 13:45:42.998983 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.997875 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 13:45:42.998983 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.998057 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c87cev9jmvkrh\"" Jul 21 13:45:42.998983 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.998370 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mf2hg\"" Jul 21 13:45:42.998983 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.998572 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 13:45:42.998983 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:42.998829 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 13:45:43.090552 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.090497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-client-ca-bundle\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.090745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.090573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-secret-metrics-server-tls\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.090745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.090621 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k7qw\" (UniqueName: \"kubernetes.io/projected/2b5027e9-e351-47a0-8e33-e8184f4ebef0-kube-api-access-4k7qw\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.090745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.090665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b5027e9-e351-47a0-8e33-e8184f4ebef0-audit-log\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.090908 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.090810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b5027e9-e351-47a0-8e33-e8184f4ebef0-metrics-server-audit-profiles\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.090908 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.090880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-secret-metrics-server-client-certs\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.091014 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.090923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b5027e9-e351-47a0-8e33-e8184f4ebef0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.192474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.191421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k7qw\" (UniqueName: \"kubernetes.io/projected/2b5027e9-e351-47a0-8e33-e8184f4ebef0-kube-api-access-4k7qw\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.192474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.191491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b5027e9-e351-47a0-8e33-e8184f4ebef0-audit-log\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.192474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.191553 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b5027e9-e351-47a0-8e33-e8184f4ebef0-metrics-server-audit-profiles\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.192474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.191612 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-secret-metrics-server-client-certs\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.192474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.191656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b5027e9-e351-47a0-8e33-e8184f4ebef0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.192474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.191700 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-client-ca-bundle\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.192474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.191727 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-secret-metrics-server-tls\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.192474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.192090 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b5027e9-e351-47a0-8e33-e8184f4ebef0-audit-log\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.196579 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.193557 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b5027e9-e351-47a0-8e33-e8184f4ebef0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.196579 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.195137 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b5027e9-e351-47a0-8e33-e8184f4ebef0-metrics-server-audit-profiles\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.196579 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.195640 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-client-ca-bundle\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.196579 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.196141 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-secret-metrics-server-tls\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.196870 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.196618 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b5027e9-e351-47a0-8e33-e8184f4ebef0-secret-metrics-server-client-certs\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.202296 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.202177 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k7qw\" (UniqueName: \"kubernetes.io/projected/2b5027e9-e351-47a0-8e33-e8184f4ebef0-kube-api-access-4k7qw\") pod \"metrics-server-7b857c966-2pwpf\" (UID: \"2b5027e9-e351-47a0-8e33-e8184f4ebef0\") " pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.311098 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.310821 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:45:43.378476 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.377093 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm"] Jul 21 13:45:43.391211 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.391171 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm"] Jul 21 13:45:43.391371 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.391303 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:43.393891 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.393865 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 13:45:43.394109 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.394092 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fngwv\"" Jul 21 13:45:43.494875 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.494708 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntjgp\" (UniqueName: \"kubernetes.io/projected/e67d0ca1-8ff4-40e6-baa1-fd307def1ff9-kube-api-access-ntjgp\") pod \"monitoring-plugin-7b7b7bc8bf-682qm\" (UID: \"e67d0ca1-8ff4-40e6-baa1-fd307def1ff9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:43.494875 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.494782 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e67d0ca1-8ff4-40e6-baa1-fd307def1ff9-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-682qm\" (UID: \"e67d0ca1-8ff4-40e6-baa1-fd307def1ff9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:43.595393 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.595304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjgp\" (UniqueName: \"kubernetes.io/projected/e67d0ca1-8ff4-40e6-baa1-fd307def1ff9-kube-api-access-ntjgp\") pod \"monitoring-plugin-7b7b7bc8bf-682qm\" (UID: \"e67d0ca1-8ff4-40e6-baa1-fd307def1ff9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:43.595393 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.595371 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e67d0ca1-8ff4-40e6-baa1-fd307def1ff9-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-682qm\" (UID: \"e67d0ca1-8ff4-40e6-baa1-fd307def1ff9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:43.600921 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.598702 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e67d0ca1-8ff4-40e6-baa1-fd307def1ff9-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-682qm\" (UID: \"e67d0ca1-8ff4-40e6-baa1-fd307def1ff9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:43.604330 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.604302 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjgp\" (UniqueName: \"kubernetes.io/projected/e67d0ca1-8ff4-40e6-baa1-fd307def1ff9-kube-api-access-ntjgp\") pod \"monitoring-plugin-7b7b7bc8bf-682qm\" (UID: \"e67d0ca1-8ff4-40e6-baa1-fd307def1ff9\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:43.647451 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.647415 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-768987b95f-fh2b8"] Jul 21 13:45:43.672668 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.672631 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-884f78984-2jh6q"] Jul 21 13:45:43.705843 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.705609 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:43.727357 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.727310 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-884f78984-2jh6q"] Jul 21 13:45:43.727565 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.727425 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.838247 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.838179 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs"] Jul 21 13:45:43.867005 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.866919 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs"] Jul 21 13:45:43.867155 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.867096 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.870183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.870154 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jzlp9\"" Jul 21 13:45:43.870498 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.870473 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 21 13:45:43.870752 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.870734 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 21 13:45:43.871358 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.871336 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 21 13:45:43.871968 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.871834 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 21 13:45:43.871968 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.871864 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 21 13:45:43.876338 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.876315 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 21 13:45:43.897754 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.897717 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-oauth-config\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.897937 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.897792 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-serving-cert\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.897937 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.897823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-trusted-ca-bundle\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.897937 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.897871 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9tk\" (UniqueName: \"kubernetes.io/projected/23e7245c-7db9-45c8-af26-00e216c4bfe0-kube-api-access-ph9tk\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.897937 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.897897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-oauth-serving-cert\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.898156 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.897942 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-config\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.898156 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.897985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-service-ca\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999024 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.998989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-config\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999211 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999039 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-federate-client-tls\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.999211 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-secret-telemeter-client\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.999211 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-metrics-client-ca\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.999211 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-telemeter-client-tls\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.999211 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-service-ca\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-oauth-config\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999261 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.999472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-serving-cert\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999339 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-trusted-ca-bundle\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999366 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9tk\" (UniqueName: \"kubernetes.io/projected/23e7245c-7db9-45c8-af26-00e216c4bfe0-kube-api-access-ph9tk\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999391 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l845l\" (UniqueName: \"kubernetes.io/projected/3042006e-8243-49e9-a4cb-311457042576-kube-api-access-l845l\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.999472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999417 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-oauth-serving-cert\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999440 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-serving-certs-ca-bundle\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.999993 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:43.999993 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999841 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-config\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:43.999993 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:43.999948 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-service-ca\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:44.000590 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.000536 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-oauth-serving-cert\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:44.000695 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.000636 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-trusted-ca-bundle\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:44.002212 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.002185 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-serving-cert\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:44.002310 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.002221 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-oauth-config\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:44.007831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.007812 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9tk\" (UniqueName: \"kubernetes.io/projected/23e7245c-7db9-45c8-af26-00e216c4bfe0-kube-api-access-ph9tk\") pod \"console-884f78984-2jh6q\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:44.040722 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.040669 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:45:44.100310 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.100280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-secret-telemeter-client\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.100438 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.100318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-metrics-client-ca\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.100438 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.100335 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-telemeter-client-tls\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.100597 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.100573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.100667 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.100649 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l845l\" (UniqueName: \"kubernetes.io/projected/3042006e-8243-49e9-a4cb-311457042576-kube-api-access-l845l\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.100719 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.100685 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-serving-certs-ca-bundle\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.100774 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.100721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.100774 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.100760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-federate-client-tls\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.101427 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.101117 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-metrics-client-ca\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.101427 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.101366 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-serving-certs-ca-bundle\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.101838 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.101811 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3042006e-8243-49e9-a4cb-311457042576-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.102899 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.102873 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.102988 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.102953 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-telemeter-client-tls\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.103036 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.103016 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-secret-telemeter-client\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.103089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.103074 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3042006e-8243-49e9-a4cb-311457042576-federate-client-tls\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.108293 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.108268 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l845l\" (UniqueName: \"kubernetes.io/projected/3042006e-8243-49e9-a4cb-311457042576-kube-api-access-l845l\") pod \"telemeter-client-77dc48c6b5-8f6bs\" (UID: \"3042006e-8243-49e9-a4cb-311457042576\") " pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.181266 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.181213 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" Jul 21 13:45:44.858195 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.858147 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:45:44.888508 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.888470 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:45:44.888698 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.888677 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:44.893444 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.893416 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 13:45:44.893594 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.893582 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 13:45:44.893718 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.893702 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 13:45:44.893799 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.893785 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vzq75\"" Jul 21 13:45:44.893919 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.893905 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 13:45:44.894034 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.894020 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 13:45:44.894086 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.894073 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 13:45:44.894329 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.893416 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ftq8l5t3tu3rb\"" Jul 21 13:45:44.894830 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.894662 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 13:45:44.894830 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.894678 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 13:45:44.895042 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.894924 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 13:45:44.895042 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.894943 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 13:45:44.895271 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.895253 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 13:45:44.895693 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.895671 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 13:45:44.897278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.897257 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 13:45:44.900011 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:44.899581 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 13:45:45.009968 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.009930 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010168 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.009994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010168 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010168 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010125 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-config\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010340 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010167 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010340 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010221 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010340 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010326 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010491 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010491 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010491 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010491 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010437 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010714 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010492 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010714 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010520 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010714 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010714 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010714 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010620 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010714 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010654 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.010714 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5rq\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-kube-api-access-8h5rq\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.011050 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.011050 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.010769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112128 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112128 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112360 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112360 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112360 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112360 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-config\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112360 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112285 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112360 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112360 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112369 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112411 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112553 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112594 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112675 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112708 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.112745 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.112732 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5rq\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-kube-api-access-8h5rq\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.113165 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.113117 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.113366 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.113345 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.115090 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.115059 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.118181 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.115358 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.118181 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.115596 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.118181 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.117923 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.118181 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.117946 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.118759 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.118495 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.121636 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.121193 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.124601 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.124510 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.125146 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.125097 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5rq\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-kube-api-access-8h5rq\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.126280 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.126219 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.126601 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.126453 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.126840 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.126793 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.126934 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.126897 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-config\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.127433 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.127390 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.127966 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.127829 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.128329 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.128259 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.128723 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.128701 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.131915 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.131872 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:45.204327 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:45.204235 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:45:52.175589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:52.175532 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm"] Jul 21 13:45:53.091320 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:53.091286 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67d0ca1_8ff4_40e6_baa1_fd307def1ff9.slice/crio-b45e173dfa62b6fd2f6dc0e74fdac9262055c8ee9c6e34b9bd93ff9e5a9331cf WatchSource:0}: Error finding container b45e173dfa62b6fd2f6dc0e74fdac9262055c8ee9c6e34b9bd93ff9e5a9331cf: Status 404 returned error can't find the container with id b45e173dfa62b6fd2f6dc0e74fdac9262055c8ee9c6e34b9bd93ff9e5a9331cf Jul 21 13:45:53.211721 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:53.211658 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b857c966-2pwpf"] Jul 21 13:45:53.237017 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:53.236988 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" event={"ID":"e67d0ca1-8ff4-40e6-baa1-fd307def1ff9","Type":"ContainerStarted","Data":"b45e173dfa62b6fd2f6dc0e74fdac9262055c8ee9c6e34b9bd93ff9e5a9331cf"} Jul 21 13:45:53.393903 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:53.393803 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5027e9_e351_47a0_8e33_e8184f4ebef0.slice/crio-5b25b395cef871311ff4f5fea9a0bcfbcd18d41c9c666d00c2434b19ef3c69b2 WatchSource:0}: Error finding container 5b25b395cef871311ff4f5fea9a0bcfbcd18d41c9c666d00c2434b19ef3c69b2: Status 404 returned error can't find the container with id 5b25b395cef871311ff4f5fea9a0bcfbcd18d41c9c666d00c2434b19ef3c69b2 Jul 21 13:45:53.552598 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:53.552556 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-884f78984-2jh6q"] Jul 21 13:45:53.567246 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:45:53.567210 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e7245c_7db9_45c8_af26_00e216c4bfe0.slice/crio-87358aa3dcfd87e1691fe262b636a1de97621bc25c4014f6303a3a625ed895ee WatchSource:0}: Error finding container 87358aa3dcfd87e1691fe262b636a1de97621bc25c4014f6303a3a625ed895ee: Status 404 returned error can't find the container with id 87358aa3dcfd87e1691fe262b636a1de97621bc25c4014f6303a3a625ed895ee Jul 21 13:45:53.816835 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:53.816694 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs"] Jul 21 13:45:53.816835 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:53.816770 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:45:54.246709 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.246651 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" event={"ID":"20c56c13-3380-4a02-9d22-7f85f7ec346c","Type":"ContainerStarted","Data":"ea39b3e243e5406e801d3f0d32208293d2d80b33c4acf7ef5fa5d353f74f8c92"} Jul 21 13:45:54.247196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.246794 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" event={"ID":"20c56c13-3380-4a02-9d22-7f85f7ec346c","Type":"ContainerStarted","Data":"01d182e69089e36a6f773eb3821bd10bf67b9eaf5f7b0a2403ad6b2095f6e5eb"} Jul 21 13:45:54.247196 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.246837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" event={"ID":"20c56c13-3380-4a02-9d22-7f85f7ec346c","Type":"ContainerStarted","Data":"c19aa1616b43b0826ed41c9b28ddbd43468cff5030bb48e2eb6837cf2446e76b"} Jul 21 13:45:54.249368 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.248925 2569 generic.go:356] "Generic (PLEG): container finished" podID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" exitCode=0 Jul 21 13:45:54.249368 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.249019 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerDied","Data":"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b"} Jul 21 13:45:54.249368 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.249084 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerStarted","Data":"bf4fbd8ec81e3d267e69451fc6d38286eb54dec6eb3bbac9f40ff3d746ea64f1"} Jul 21 13:45:54.251612 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.251492 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r9rll" event={"ID":"c859f694-6549-4863-8b28-9e97b2b52063","Type":"ContainerStarted","Data":"88f0d81eb22f978e71e11d5faddd3283a25e640993b2801861a653b92093ecd0"} Jul 21 13:45:54.251829 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.251779 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:45:54.253899 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.253873 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768987b95f-fh2b8" event={"ID":"58be5d89-df08-45b5-8fed-5afc5a84f585","Type":"ContainerStarted","Data":"3182ab217345edc4cbd9e7d728777974e6f4dc8d406b9dfd3d406ec6dd979dd3"} Jul 21 13:45:54.256233 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.256207 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" event={"ID":"2b5027e9-e351-47a0-8e33-e8184f4ebef0","Type":"ContainerStarted","Data":"5b25b395cef871311ff4f5fea9a0bcfbcd18d41c9c666d00c2434b19ef3c69b2"} Jul 21 13:45:54.258363 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.258336 2569 generic.go:356] "Generic (PLEG): container finished" podID="9063bc3f-f3cb-4946-9841-b47b6503b7b4" containerID="1d4914295e3a25fef46f9b2c9241411cb9e4359febaa7c3f5cadb9ebf5622888" exitCode=0 Jul 21 13:45:54.258461 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.258414 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrdw" event={"ID":"9063bc3f-f3cb-4946-9841-b47b6503b7b4","Type":"ContainerDied","Data":"1d4914295e3a25fef46f9b2c9241411cb9e4359febaa7c3f5cadb9ebf5622888"} Jul 21 13:45:54.263963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.263938 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" event={"ID":"1e67cdf6-b024-4463-879e-b95846974683","Type":"ContainerStarted","Data":"486fba6d0c648aca23e39e0d17e7f1d54afd72d9f64524ba5e66412a72423810"} Jul 21 13:45:54.264077 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.263993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" event={"ID":"1e67cdf6-b024-4463-879e-b95846974683","Type":"ContainerStarted","Data":"dfd9b17d7e68022a3d3af4a76ff4ffa60dd734e33863ef719a380e9bc99b971e"} Jul 21 13:45:54.264077 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.264020 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" event={"ID":"1e67cdf6-b024-4463-879e-b95846974683","Type":"ContainerStarted","Data":"773eff0839db87dbc5da1f8e74f0a2e62924b6da07c3fbf8102c23ee64ec790a"} Jul 21 13:45:54.267909 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.267846 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" event={"ID":"3042006e-8243-49e9-a4cb-311457042576","Type":"ContainerStarted","Data":"1006a18c8df32da577d2f9b9f3d024340f4e263d1c954404245ba9e775580823"} Jul 21 13:45:54.270439 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.270415 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ztpvd" event={"ID":"79c9e89d-d287-46a8-b4fe-eeb2e8a31223","Type":"ContainerStarted","Data":"dcefd646515073c6d71d8014584d9a255d918dd333f8c4b5bd0fe1049b064166"} Jul 21 13:45:54.273013 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.272989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8f77dd4d-5hzf4" event={"ID":"4790404b-c94b-40f6-9ffc-932b6798c742","Type":"ContainerStarted","Data":"93b8128f23458bc2b5187cbb1b87e895479f4f3626b1cb87383d62f91e893471"} Jul 21 13:45:54.281277 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.281222 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" event={"ID":"ba604b91-8658-4512-93f3-724667407ea1","Type":"ContainerStarted","Data":"f13ad558aacd62b2bdc022c4cc861d20d6e6e994b298f604a34da07eda82d235"} Jul 21 13:45:54.285215 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.285125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-884f78984-2jh6q" event={"ID":"23e7245c-7db9-45c8-af26-00e216c4bfe0","Type":"ContainerStarted","Data":"e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de"} Jul 21 13:45:54.285215 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.285162 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-884f78984-2jh6q" event={"ID":"23e7245c-7db9-45c8-af26-00e216c4bfe0","Type":"ContainerStarted","Data":"87358aa3dcfd87e1691fe262b636a1de97621bc25c4014f6303a3a625ed895ee"} Jul 21 13:45:54.288607 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.288508 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82f8j" event={"ID":"6fcd41c3-839c-44d9-8085-84ce0f5c5086","Type":"ContainerStarted","Data":"2e262d13c407b70a12acef5621cccd4752aca9e3b25fd0a15b4f192ab28482e9"} Jul 21 13:45:54.291973 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.291877 2569 generic.go:356] "Generic (PLEG): container finished" podID="31cfef40-9134-4b59-887c-097dadf6a797" containerID="4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae" exitCode=0 Jul 21 13:45:54.291973 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.291910 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerDied","Data":"4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae"} Jul 21 13:45:54.294248 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.294201 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r9rll" podStartSLOduration=70.125525066 podStartE2EDuration="1m26.294186741s" totalImagesPullingTime="16.168661675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:54.294167604 +0000 UTC m=+87.332237526" watchObservedRunningTime="2026-07-21 13:45:54.294186741 +0000 UTC m=+87.332256658" Jul 21 13:45:54.313066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.312741 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-g896z" podStartSLOduration=3.831294785 podStartE2EDuration="16.312727228s" totalImagesPullingTime="12.481432443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:54.312101306 +0000 UTC m=+87.350171228" watchObservedRunningTime="2026-07-21 13:45:54.312727228 +0000 UTC m=+87.350797148" Jul 21 13:45:54.330114 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.330038 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-768987b95f-fh2b8" podStartSLOduration=11.367965792 podStartE2EDuration="28.330017504s" totalImagesPullingTime="16.962051712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:54.329967323 +0000 UTC m=+87.368037277" watchObservedRunningTime="2026-07-21 13:45:54.330017504 +0000 UTC m=+87.368087426" Jul 21 13:45:54.367995 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.367763 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ztpvd" podStartSLOduration=69.248026636 podStartE2EDuration="1m25.367741675s" totalImagesPullingTime="16.119715039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:54.367707306 +0000 UTC m=+87.405777227" watchObservedRunningTime="2026-07-21 13:45:54.367741675 +0000 UTC m=+87.405811599" Jul 21 13:45:54.388475 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.388426 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c8f77dd4d-5hzf4" podStartSLOduration=4.70138138 podStartE2EDuration="19.388411186s" totalImagesPullingTime="14.687029806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:54.387775303 +0000 UTC m=+87.425845229" watchObservedRunningTime="2026-07-21 13:45:54.388411186 +0000 UTC m=+87.426481129" Jul 21 13:45:54.431058 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.430779 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dtcsn" podStartSLOduration=4.419937354 podStartE2EDuration="16.430762314s" totalImagesPullingTime="12.01082496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:54.430003287 +0000 UTC m=+87.468073208" watchObservedRunningTime="2026-07-21 13:45:54.430762314 +0000 UTC m=+87.468832234" Jul 21 13:45:54.448241 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.448198 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-82f8j" podStartSLOduration=2.509241327 podStartE2EDuration="38.448184s" totalImagesPullingTime="35.938942673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:45:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:54.446391235 +0000 UTC m=+87.484461155" watchObservedRunningTime="2026-07-21 13:45:54.448184 +0000 UTC m=+87.486253921" Jul 21 13:45:54.465277 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:54.465227 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-884f78984-2jh6q" podStartSLOduration=11.465212525 podStartE2EDuration="11.465212525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:54.463836267 +0000 UTC m=+87.501906212" watchObservedRunningTime="2026-07-21 13:45:54.465212525 +0000 UTC m=+87.503282446" Jul 21 13:45:55.301143 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:55.301079 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrdw" event={"ID":"9063bc3f-f3cb-4946-9841-b47b6503b7b4","Type":"ContainerStarted","Data":"b1a0b27ffdaf9164e4e3aa329b7fb007eb3dd1cd392645dba63c1b513a22eb36"} Jul 21 13:45:55.301727 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:55.301169 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xgrdw" event={"ID":"9063bc3f-f3cb-4946-9841-b47b6503b7b4","Type":"ContainerStarted","Data":"0848cd5667c9978fc6708c512e5b334e98627c447179cf64342ba526b45f2940"} Jul 21 13:45:55.322205 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:55.322158 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xgrdw" podStartSLOduration=4.213622997 podStartE2EDuration="17.32214421s" totalImagesPullingTime="12.947592086s" totalInitContainerRuntime="160.929127ms" isStatefulPod=true podCreationTimestamp="2026-07-21 13:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:55.321226896 +0000 UTC m=+88.359296818" watchObservedRunningTime="2026-07-21 13:45:55.32214421 +0000 UTC m=+88.360214131" Jul 21 13:45:55.360010 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:55.359963 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:55.360168 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:55.360038 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:55.365821 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:55.365796 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:56.311734 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:56.311652 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:45:56.760994 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:56.760926 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:45:58.325567 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:58.325518 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" event={"ID":"2b5027e9-e351-47a0-8e33-e8184f4ebef0","Type":"ContainerStarted","Data":"ff0704b61f4a52ddec02c16c6e0f42a2b9a6c077953be160cc0441e066f5c786"} Jul 21 13:45:58.327602 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:58.327566 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" event={"ID":"3042006e-8243-49e9-a4cb-311457042576","Type":"ContainerStarted","Data":"906050cb50f2e31e7a6ffcf550f87ac51e19ecf9bf3e5613f935ef4ce0cf59c1"} Jul 21 13:45:58.329301 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:58.329274 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" event={"ID":"e67d0ca1-8ff4-40e6-baa1-fd307def1ff9","Type":"ContainerStarted","Data":"73283646b9cbe1f9511ba7031f4d07fb600ea971f41203a07c87a03325615cb5"} Jul 21 13:45:58.329481 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:58.329466 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:58.332326 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:58.332302 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" event={"ID":"20c56c13-3380-4a02-9d22-7f85f7ec346c","Type":"ContainerStarted","Data":"5439ac523e203b697ae1cf0cd4bbbdf685e5988a40758fab71742e8dde26d2db"} Jul 21 13:45:58.335327 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:58.335304 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" Jul 21 13:45:58.351620 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:58.351565 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" podStartSLOduration=12.69486847 podStartE2EDuration="16.351526317s" totalImagesPullingTime="3.656657847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:58.348874071 +0000 UTC m=+91.386943993" watchObservedRunningTime="2026-07-21 13:45:58.351526317 +0000 UTC m=+91.389596240" Jul 21 13:45:58.364931 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:45:58.364879 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-682qm" podStartSLOduration=11.492440943 podStartE2EDuration="15.364864299s" totalImagesPullingTime="3.872423356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:45:58.363725435 +0000 UTC m=+91.401795357" watchObservedRunningTime="2026-07-21 13:45:58.364864299 +0000 UTC m=+91.402934249" Jul 21 13:46:00.343713 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.343665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" event={"ID":"3042006e-8243-49e9-a4cb-311457042576","Type":"ContainerStarted","Data":"39d4f8338aa3ae67c431bb27f2c12fcc685fd1ee857e5c6ecfa1ad3c12686945"} Jul 21 13:46:00.344047 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.343760 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" event={"ID":"3042006e-8243-49e9-a4cb-311457042576","Type":"ContainerStarted","Data":"d4c3dba65387e265a7a1906c79e240c6786643df766b604b9968c68972d98f98"} Jul 21 13:46:00.347233 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.347203 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerStarted","Data":"4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be"} Jul 21 13:46:00.347351 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.347290 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerStarted","Data":"9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868"} Jul 21 13:46:00.347351 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.347330 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerStarted","Data":"2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363"} Jul 21 13:46:00.347466 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.347355 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerStarted","Data":"ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0"} Jul 21 13:46:00.347466 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.347399 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerStarted","Data":"9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d"} Jul 21 13:46:00.347466 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.347426 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerStarted","Data":"8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0"} Jul 21 13:46:00.350323 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.350293 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" event={"ID":"20c56c13-3380-4a02-9d22-7f85f7ec346c","Type":"ContainerStarted","Data":"3ac00e11aff19c10cd1b7f7e6b496d59efbf3d1bfbf593c3cc3ad9694cf2cab8"} Jul 21 13:46:00.350427 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.350350 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" event={"ID":"20c56c13-3380-4a02-9d22-7f85f7ec346c","Type":"ContainerStarted","Data":"1258f0426d77edd00cd78290bad17279282613780d9373c9c443b3174bb205e3"} Jul 21 13:46:00.350501 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.350479 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:46:00.354010 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.353870 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerStarted","Data":"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f"} Jul 21 13:46:00.354010 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.353920 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerStarted","Data":"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027"} Jul 21 13:46:00.354010 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.353953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerStarted","Data":"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391"} Jul 21 13:46:00.354010 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.353978 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerStarted","Data":"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b"} Jul 21 13:46:00.357112 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.357095 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" Jul 21 13:46:00.368355 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.368306 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-77dc48c6b5-8f6bs" podStartSLOduration=13.468606976 podStartE2EDuration="17.368288636s" totalImagesPullingTime="3.89968166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:46:00.366425823 +0000 UTC m=+93.404495744" watchObservedRunningTime="2026-07-21 13:46:00.368288636 +0000 UTC m=+93.406358559" Jul 21 13:46:00.398646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.398597 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.174283504 podStartE2EDuration="21.398577098s" totalImagesPullingTime="17.211729075s" totalInitContainerRuntime="12.564519ms" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:46:00.397150628 +0000 UTC m=+93.435220550" watchObservedRunningTime="2026-07-21 13:46:00.398577098 +0000 UTC m=+93.436647020" Jul 21 13:46:00.426329 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:00.426066 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5656dbdcdc-njbbd" podStartSLOduration=4.611593868 podStartE2EDuration="20.426044864s" totalImagesPullingTime="15.814450996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:46:00.425080502 +0000 UTC m=+93.463150437" watchObservedRunningTime="2026-07-21 13:46:00.426044864 +0000 UTC m=+93.464114786" Jul 21 13:46:01.084042 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:01.084004 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-884f78984-2jh6q"] Jul 21 13:46:01.363726 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:01.363632 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerStarted","Data":"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb"} Jul 21 13:46:01.363726 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:01.363706 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerStarted","Data":"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f"} Jul 21 13:46:01.401465 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:01.401388 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=11.96030187 podStartE2EDuration="17.401366916s" totalImagesPullingTime="5.418780198s" totalInitContainerRuntime="22.284848ms" isStatefulPod=false podCreationTimestamp="2026-07-21 13:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:46:01.399459648 +0000 UTC m=+94.437529582" watchObservedRunningTime="2026-07-21 13:46:01.401366916 +0000 UTC m=+94.439436842" Jul 21 13:46:03.311646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:03.311613 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:46:03.312002 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:03.311702 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:46:04.041357 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:04.041320 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:46:05.204643 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:05.204613 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:46:12.305437 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:12.305403 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c8f77dd4d-5hzf4"] Jul 21 13:46:19.283578 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.283508 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-768987b95f-fh2b8" podUID="58be5d89-df08-45b5-8fed-5afc5a84f585" containerName="console" containerID="cri-o://3182ab217345edc4cbd9e7d728777974e6f4dc8d406b9dfd3d406ec6dd979dd3" gracePeriod=15 Jul 21 13:46:19.456288 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.456262 2569 generic.go:356] "Generic (PLEG): container finished" podID="58be5d89-df08-45b5-8fed-5afc5a84f585" containerID="3182ab217345edc4cbd9e7d728777974e6f4dc8d406b9dfd3d406ec6dd979dd3" exitCode=0 Jul 21 13:46:19.456402 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.456303 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768987b95f-fh2b8" event={"ID":"58be5d89-df08-45b5-8fed-5afc5a84f585","Type":"ContainerDied","Data":"3182ab217345edc4cbd9e7d728777974e6f4dc8d406b9dfd3d406ec6dd979dd3"} Jul 21 13:46:19.557439 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.557387 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:46:19.660679 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.660645 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qvtq\" (UniqueName: \"kubernetes.io/projected/58be5d89-df08-45b5-8fed-5afc5a84f585-kube-api-access-7qvtq\") pod \"58be5d89-df08-45b5-8fed-5afc5a84f585\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " Jul 21 13:46:19.660845 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.660684 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-serving-cert\") pod \"58be5d89-df08-45b5-8fed-5afc5a84f585\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " Jul 21 13:46:19.660845 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.660715 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-oauth-config\") pod \"58be5d89-df08-45b5-8fed-5afc5a84f585\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " Jul 21 13:46:19.660845 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.660741 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-oauth-serving-cert\") pod \"58be5d89-df08-45b5-8fed-5afc5a84f585\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " Jul 21 13:46:19.660845 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.660769 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-service-ca\") pod \"58be5d89-df08-45b5-8fed-5afc5a84f585\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " Jul 21 13:46:19.660845 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.660820 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-console-config\") pod \"58be5d89-df08-45b5-8fed-5afc5a84f585\" (UID: \"58be5d89-df08-45b5-8fed-5afc5a84f585\") " Jul 21 13:46:19.661278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.661247 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-service-ca" (OuterVolumeSpecName: "service-ca") pod "58be5d89-df08-45b5-8fed-5afc5a84f585" (UID: "58be5d89-df08-45b5-8fed-5afc5a84f585"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:19.661278 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.661258 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "58be5d89-df08-45b5-8fed-5afc5a84f585" (UID: "58be5d89-df08-45b5-8fed-5afc5a84f585"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:19.661402 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.661243 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-console-config" (OuterVolumeSpecName: "console-config") pod "58be5d89-df08-45b5-8fed-5afc5a84f585" (UID: "58be5d89-df08-45b5-8fed-5afc5a84f585"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:19.663054 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.663027 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "58be5d89-df08-45b5-8fed-5afc5a84f585" (UID: "58be5d89-df08-45b5-8fed-5afc5a84f585"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:46:19.663408 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.663385 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58be5d89-df08-45b5-8fed-5afc5a84f585-kube-api-access-7qvtq" (OuterVolumeSpecName: "kube-api-access-7qvtq") pod "58be5d89-df08-45b5-8fed-5afc5a84f585" (UID: "58be5d89-df08-45b5-8fed-5afc5a84f585"). InnerVolumeSpecName "kube-api-access-7qvtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:46:19.663483 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.663396 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "58be5d89-df08-45b5-8fed-5afc5a84f585" (UID: "58be5d89-df08-45b5-8fed-5afc5a84f585"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:46:19.761687 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.761646 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvtq\" (UniqueName: \"kubernetes.io/projected/58be5d89-df08-45b5-8fed-5afc5a84f585-kube-api-access-7qvtq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:19.761687 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.761675 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:19.761687 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.761684 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58be5d89-df08-45b5-8fed-5afc5a84f585-console-oauth-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:19.761687 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.761693 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-oauth-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:19.761687 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.761702 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:19.761991 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:19.761710 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58be5d89-df08-45b5-8fed-5afc5a84f585-console-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:20.461387 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:20.461351 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768987b95f-fh2b8" event={"ID":"58be5d89-df08-45b5-8fed-5afc5a84f585","Type":"ContainerDied","Data":"ffce84705fe6bf5559fda42cea54347a851f1d82fffefd6e30143bb2b56d655e"} Jul 21 13:46:20.461836 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:20.461396 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768987b95f-fh2b8" Jul 21 13:46:20.461836 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:20.461433 2569 scope.go:117] "RemoveContainer" containerID="3182ab217345edc4cbd9e7d728777974e6f4dc8d406b9dfd3d406ec6dd979dd3" Jul 21 13:46:20.483615 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:20.483583 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-768987b95f-fh2b8"] Jul 21 13:46:20.487226 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:20.487206 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-768987b95f-fh2b8"] Jul 21 13:46:20.794638 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:20.794607 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58be5d89-df08-45b5-8fed-5afc5a84f585" path="/var/lib/kubelet/pods/58be5d89-df08-45b5-8fed-5afc5a84f585/volumes" Jul 21 13:46:23.317008 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:23.316980 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:46:23.320866 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:23.320845 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b857c966-2pwpf" Jul 21 13:46:25.303475 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:25.303444 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r9rll" Jul 21 13:46:26.104271 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.104218 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-884f78984-2jh6q" podUID="23e7245c-7db9-45c8-af26-00e216c4bfe0" containerName="console" containerID="cri-o://e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de" gracePeriod=15 Jul 21 13:46:26.373358 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.373286 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:46:26.490634 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.490598 2569 generic.go:356] "Generic (PLEG): container finished" podID="23e7245c-7db9-45c8-af26-00e216c4bfe0" containerID="e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de" exitCode=0 Jul 21 13:46:26.490765 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.490646 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-884f78984-2jh6q" Jul 21 13:46:26.490765 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.490682 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-884f78984-2jh6q" event={"ID":"23e7245c-7db9-45c8-af26-00e216c4bfe0","Type":"ContainerDied","Data":"e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de"} Jul 21 13:46:26.490765 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.490734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-884f78984-2jh6q" event={"ID":"23e7245c-7db9-45c8-af26-00e216c4bfe0","Type":"ContainerDied","Data":"87358aa3dcfd87e1691fe262b636a1de97621bc25c4014f6303a3a625ed895ee"} Jul 21 13:46:26.490765 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.490757 2569 scope.go:117] "RemoveContainer" containerID="e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de" Jul 21 13:46:26.499009 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.498987 2569 scope.go:117] "RemoveContainer" containerID="e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de" Jul 21 13:46:26.499264 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:46:26.499240 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de\": container with ID starting with e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de not found: ID does not exist" containerID="e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de" Jul 21 13:46:26.499337 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.499272 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de"} err="failed to get container status \"e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de\": rpc error: code = NotFound desc = could not find container \"e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de\": container with ID starting with e3d26182542f8d2bd9001c8fd99f04ab86663b5b0c5512a1a38640239a4926de not found: ID does not exist" Jul 21 13:46:26.523567 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.523528 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-oauth-config\") pod \"23e7245c-7db9-45c8-af26-00e216c4bfe0\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " Jul 21 13:46:26.523650 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.523580 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph9tk\" (UniqueName: \"kubernetes.io/projected/23e7245c-7db9-45c8-af26-00e216c4bfe0-kube-api-access-ph9tk\") pod \"23e7245c-7db9-45c8-af26-00e216c4bfe0\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " Jul 21 13:46:26.523725 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.523713 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-config\") pod \"23e7245c-7db9-45c8-af26-00e216c4bfe0\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " Jul 21 13:46:26.523797 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.523784 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-service-ca\") pod \"23e7245c-7db9-45c8-af26-00e216c4bfe0\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " Jul 21 13:46:26.523845 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.523817 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-oauth-serving-cert\") pod \"23e7245c-7db9-45c8-af26-00e216c4bfe0\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " Jul 21 13:46:26.523890 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.523856 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-trusted-ca-bundle\") pod \"23e7245c-7db9-45c8-af26-00e216c4bfe0\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " Jul 21 13:46:26.523946 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.523888 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-serving-cert\") pod \"23e7245c-7db9-45c8-af26-00e216c4bfe0\" (UID: \"23e7245c-7db9-45c8-af26-00e216c4bfe0\") " Jul 21 13:46:26.524183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.524149 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-config" (OuterVolumeSpecName: "console-config") pod "23e7245c-7db9-45c8-af26-00e216c4bfe0" (UID: "23e7245c-7db9-45c8-af26-00e216c4bfe0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:26.524260 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.524195 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-service-ca" (OuterVolumeSpecName: "service-ca") pod "23e7245c-7db9-45c8-af26-00e216c4bfe0" (UID: "23e7245c-7db9-45c8-af26-00e216c4bfe0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:26.524260 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.524204 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "23e7245c-7db9-45c8-af26-00e216c4bfe0" (UID: "23e7245c-7db9-45c8-af26-00e216c4bfe0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:26.524365 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.524277 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "23e7245c-7db9-45c8-af26-00e216c4bfe0" (UID: "23e7245c-7db9-45c8-af26-00e216c4bfe0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:26.525727 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.525706 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e7245c-7db9-45c8-af26-00e216c4bfe0-kube-api-access-ph9tk" (OuterVolumeSpecName: "kube-api-access-ph9tk") pod "23e7245c-7db9-45c8-af26-00e216c4bfe0" (UID: "23e7245c-7db9-45c8-af26-00e216c4bfe0"). InnerVolumeSpecName "kube-api-access-ph9tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:46:26.525820 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.525799 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "23e7245c-7db9-45c8-af26-00e216c4bfe0" (UID: "23e7245c-7db9-45c8-af26-00e216c4bfe0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:46:26.526079 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.526059 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "23e7245c-7db9-45c8-af26-00e216c4bfe0" (UID: "23e7245c-7db9-45c8-af26-00e216c4bfe0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:46:26.625582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.625493 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:26.625582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.625524 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-oauth-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:26.625582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.625534 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:26.625582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.625561 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:26.625582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.625570 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-oauth-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:26.625582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.625580 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9tk\" (UniqueName: \"kubernetes.io/projected/23e7245c-7db9-45c8-af26-00e216c4bfe0-kube-api-access-ph9tk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:26.625582 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.625589 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23e7245c-7db9-45c8-af26-00e216c4bfe0-console-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:26.814457 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.814429 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-884f78984-2jh6q"] Jul 21 13:46:26.818243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:26.818219 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-884f78984-2jh6q"] Jul 21 13:46:28.795030 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:28.794997 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23e7245c-7db9-45c8-af26-00e216c4bfe0" path="/var/lib/kubelet/pods/23e7245c-7db9-45c8-af26-00e216c4bfe0/volumes" Jul 21 13:46:37.333502 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.333455 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c8f77dd4d-5hzf4" podUID="4790404b-c94b-40f6-9ffc-932b6798c742" containerName="console" containerID="cri-o://93b8128f23458bc2b5187cbb1b87e895479f4f3626b1cb87383d62f91e893471" gracePeriod=15 Jul 21 13:46:37.539932 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.539897 2569 generic.go:356] "Generic (PLEG): container finished" podID="4790404b-c94b-40f6-9ffc-932b6798c742" containerID="93b8128f23458bc2b5187cbb1b87e895479f4f3626b1cb87383d62f91e893471" exitCode=0 Jul 21 13:46:37.540154 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.540135 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8f77dd4d-5hzf4" event={"ID":"4790404b-c94b-40f6-9ffc-932b6798c742","Type":"ContainerDied","Data":"93b8128f23458bc2b5187cbb1b87e895479f4f3626b1cb87383d62f91e893471"} Jul 21 13:46:37.613357 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.613288 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:46:37.701070 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701042 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-trusted-ca-bundle\") pod \"4790404b-c94b-40f6-9ffc-932b6798c742\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " Jul 21 13:46:37.701222 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701079 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz2kg\" (UniqueName: \"kubernetes.io/projected/4790404b-c94b-40f6-9ffc-932b6798c742-kube-api-access-wz2kg\") pod \"4790404b-c94b-40f6-9ffc-932b6798c742\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " Jul 21 13:46:37.701222 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701100 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-console-config\") pod \"4790404b-c94b-40f6-9ffc-932b6798c742\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " Jul 21 13:46:37.701222 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701157 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-service-ca\") pod \"4790404b-c94b-40f6-9ffc-932b6798c742\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " Jul 21 13:46:37.701222 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701173 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-serving-cert\") pod \"4790404b-c94b-40f6-9ffc-932b6798c742\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " Jul 21 13:46:37.701222 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701189 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-oauth-serving-cert\") pod \"4790404b-c94b-40f6-9ffc-932b6798c742\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " Jul 21 13:46:37.701222 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701209 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-oauth-config\") pod \"4790404b-c94b-40f6-9ffc-932b6798c742\" (UID: \"4790404b-c94b-40f6-9ffc-932b6798c742\") " Jul 21 13:46:37.701558 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701517 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4790404b-c94b-40f6-9ffc-932b6798c742" (UID: "4790404b-c94b-40f6-9ffc-932b6798c742"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:37.701704 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701676 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-console-config" (OuterVolumeSpecName: "console-config") pod "4790404b-c94b-40f6-9ffc-932b6798c742" (UID: "4790404b-c94b-40f6-9ffc-932b6798c742"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:37.701704 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701688 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-service-ca" (OuterVolumeSpecName: "service-ca") pod "4790404b-c94b-40f6-9ffc-932b6798c742" (UID: "4790404b-c94b-40f6-9ffc-932b6798c742"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:37.701825 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.701712 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4790404b-c94b-40f6-9ffc-932b6798c742" (UID: "4790404b-c94b-40f6-9ffc-932b6798c742"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:46:37.703303 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.703271 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4790404b-c94b-40f6-9ffc-932b6798c742-kube-api-access-wz2kg" (OuterVolumeSpecName: "kube-api-access-wz2kg") pod "4790404b-c94b-40f6-9ffc-932b6798c742" (UID: "4790404b-c94b-40f6-9ffc-932b6798c742"). InnerVolumeSpecName "kube-api-access-wz2kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:46:37.703432 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.703414 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4790404b-c94b-40f6-9ffc-932b6798c742" (UID: "4790404b-c94b-40f6-9ffc-932b6798c742"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:46:37.703484 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.703468 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4790404b-c94b-40f6-9ffc-932b6798c742" (UID: "4790404b-c94b-40f6-9ffc-932b6798c742"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:46:37.802238 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.802206 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:37.802238 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.802232 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:37.802238 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.802242 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-oauth-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:37.802445 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.802251 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4790404b-c94b-40f6-9ffc-932b6798c742-console-oauth-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:37.802445 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.802262 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:37.802445 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.802270 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz2kg\" (UniqueName: \"kubernetes.io/projected/4790404b-c94b-40f6-9ffc-932b6798c742-kube-api-access-wz2kg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:37.802445 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:37.802279 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4790404b-c94b-40f6-9ffc-932b6798c742-console-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:46:38.545263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:38.545237 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8f77dd4d-5hzf4" Jul 21 13:46:38.545734 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:38.545236 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8f77dd4d-5hzf4" event={"ID":"4790404b-c94b-40f6-9ffc-932b6798c742","Type":"ContainerDied","Data":"6d93ec6ce96f6a08670011a17d4ab4dca5c3c45c2d33996e9315d0068a0a7ce7"} Jul 21 13:46:38.545734 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:38.545388 2569 scope.go:117] "RemoveContainer" containerID="93b8128f23458bc2b5187cbb1b87e895479f4f3626b1cb87383d62f91e893471" Jul 21 13:46:38.569503 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:38.569476 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c8f77dd4d-5hzf4"] Jul 21 13:46:38.575625 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:38.575596 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c8f77dd4d-5hzf4"] Jul 21 13:46:38.795455 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:38.795374 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4790404b-c94b-40f6-9ffc-932b6798c742" path="/var/lib/kubelet/pods/4790404b-c94b-40f6-9ffc-932b6798c742/volumes" Jul 21 13:46:45.205404 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:45.205368 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:46:45.224067 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:45.224042 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:46:45.587220 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:45.587194 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:46:59.024318 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.024279 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:46:59.024947 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.024716 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="alertmanager" containerID="cri-o://8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0" gracePeriod=120 Jul 21 13:46:59.024947 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.024809 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="config-reloader" containerID="cri-o://9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d" gracePeriod=120 Jul 21 13:46:59.024947 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.024794 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy-metric" containerID="cri-o://9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868" gracePeriod=120 Jul 21 13:46:59.024947 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.024848 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy" containerID="cri-o://2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363" gracePeriod=120 Jul 21 13:46:59.024947 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.024810 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy-web" containerID="cri-o://ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0" gracePeriod=120 Jul 21 13:46:59.024947 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.024939 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="prom-label-proxy" containerID="cri-o://4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be" gracePeriod=120 Jul 21 13:46:59.629978 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.629943 2569 generic.go:356] "Generic (PLEG): container finished" podID="31cfef40-9134-4b59-887c-097dadf6a797" containerID="4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be" exitCode=0 Jul 21 13:46:59.629978 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.629968 2569 generic.go:356] "Generic (PLEG): container finished" podID="31cfef40-9134-4b59-887c-097dadf6a797" containerID="9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868" exitCode=0 Jul 21 13:46:59.629978 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.629974 2569 generic.go:356] "Generic (PLEG): container finished" podID="31cfef40-9134-4b59-887c-097dadf6a797" containerID="2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363" exitCode=0 Jul 21 13:46:59.629978 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.629980 2569 generic.go:356] "Generic (PLEG): container finished" podID="31cfef40-9134-4b59-887c-097dadf6a797" containerID="9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d" exitCode=0 Jul 21 13:46:59.629978 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.629985 2569 generic.go:356] "Generic (PLEG): container finished" podID="31cfef40-9134-4b59-887c-097dadf6a797" containerID="8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0" exitCode=0 Jul 21 13:46:59.630256 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.630010 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerDied","Data":"4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be"} Jul 21 13:46:59.630256 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.630064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerDied","Data":"9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868"} Jul 21 13:46:59.630256 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.630090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerDied","Data":"2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363"} Jul 21 13:46:59.630256 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.630112 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerDied","Data":"9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d"} Jul 21 13:46:59.630256 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:46:59.630135 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerDied","Data":"8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0"} Jul 21 13:47:00.276760 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.276734 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.391661 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391582 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k288f\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-kube-api-access-k288f\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.391661 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391632 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-web\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.391867 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391666 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-cluster-tls-config\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.391867 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391778 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-metrics-client-ca\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.391867 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391831 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-metric\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392026 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391885 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-main-tls\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392026 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391929 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-tls-assets\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392026 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391968 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-config-volume\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392026 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.391994 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-config-out\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392218 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.392030 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-web-config\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392218 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.392062 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-trusted-ca-bundle\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392218 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.392091 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392218 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.392130 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-main-db\") pod \"31cfef40-9134-4b59-887c-097dadf6a797\" (UID: \"31cfef40-9134-4b59-887c-097dadf6a797\") " Jul 21 13:47:00.392218 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.392143 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:00.392437 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.392413 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-metrics-client-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.392999 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.392702 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:00.392999 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.392728 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:47:00.395143 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.395116 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-config-volume" (OuterVolumeSpecName: "config-volume") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:00.395508 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.395478 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:00.395708 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.395678 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:00.395831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.395799 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-kube-api-access-k288f" (OuterVolumeSpecName: "kube-api-access-k288f") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "kube-api-access-k288f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:47:00.395887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.395811 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-config-out" (OuterVolumeSpecName: "config-out") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:47:00.395887 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.395827 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:00.396712 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.396685 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:47:00.396979 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.396960 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:00.399605 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.399441 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:00.405963 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.405944 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-web-config" (OuterVolumeSpecName: "web-config") pod "31cfef40-9134-4b59-887c-097dadf6a797" (UID: "31cfef40-9134-4b59-887c-097dadf6a797"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.493886 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-cluster-tls-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.493919 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.493938 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-main-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.493952 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-tls-assets\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.493972 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.493984 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-config-out\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.493998 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-web-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.494010 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.494030 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.494044 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31cfef40-9134-4b59-887c-097dadf6a797-alertmanager-main-db\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.494056 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k288f\" (UniqueName: \"kubernetes.io/projected/31cfef40-9134-4b59-887c-097dadf6a797-kube-api-access-k288f\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.498089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.494070 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31cfef40-9134-4b59-887c-097dadf6a797-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:00.636946 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.636911 2569 generic.go:356] "Generic (PLEG): container finished" podID="31cfef40-9134-4b59-887c-097dadf6a797" containerID="ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0" exitCode=0 Jul 21 13:47:00.637108 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.636993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerDied","Data":"ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0"} Jul 21 13:47:00.637108 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.637009 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.637108 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.637095 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31cfef40-9134-4b59-887c-097dadf6a797","Type":"ContainerDied","Data":"8273bf1e41d681eb8a46e8c48f33828a85b4618a5ba99f7fcc5d90911d6cd94f"} Jul 21 13:47:00.637227 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.637144 2569 scope.go:117] "RemoveContainer" containerID="4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be" Jul 21 13:47:00.644486 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.644470 2569 scope.go:117] "RemoveContainer" containerID="9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868" Jul 21 13:47:00.651209 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.651192 2569 scope.go:117] "RemoveContainer" containerID="2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363" Jul 21 13:47:00.657751 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.657736 2569 scope.go:117] "RemoveContainer" containerID="ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0" Jul 21 13:47:00.664244 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.664228 2569 scope.go:117] "RemoveContainer" containerID="9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d" Jul 21 13:47:00.670796 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.670782 2569 scope.go:117] "RemoveContainer" containerID="8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0" Jul 21 13:47:00.677341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.677323 2569 scope.go:117] "RemoveContainer" containerID="4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae" Jul 21 13:47:00.683636 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.683618 2569 scope.go:117] "RemoveContainer" containerID="4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be" Jul 21 13:47:00.683881 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:00.683863 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be\": container with ID starting with 4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be not found: ID does not exist" containerID="4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be" Jul 21 13:47:00.683944 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.683886 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be"} err="failed to get container status \"4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be\": rpc error: code = NotFound desc = could not find container \"4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be\": container with ID starting with 4073f837715bd0f3f176c7abc6b339bb2c46d16d69901ba7d09fb2e1f36f29be not found: ID does not exist" Jul 21 13:47:00.683944 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.683903 2569 scope.go:117] "RemoveContainer" containerID="9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868" Jul 21 13:47:00.684149 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:00.684132 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868\": container with ID starting with 9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868 not found: ID does not exist" containerID="9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868" Jul 21 13:47:00.684192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.684155 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868"} err="failed to get container status \"9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868\": rpc error: code = NotFound desc = could not find container \"9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868\": container with ID starting with 9749be4182d97d30bca1459b6369fb17215972dcf889be82edacc34e9a568868 not found: ID does not exist" Jul 21 13:47:00.684192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.684184 2569 scope.go:117] "RemoveContainer" containerID="2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363" Jul 21 13:47:00.684408 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:00.684390 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363\": container with ID starting with 2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363 not found: ID does not exist" containerID="2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363" Jul 21 13:47:00.684455 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.684412 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363"} err="failed to get container status \"2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363\": rpc error: code = NotFound desc = could not find container \"2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363\": container with ID starting with 2fc3a9240b7e1d2feac7850e3c847b685a21b47e9f744bbc8bdbdd53c0a6d363 not found: ID does not exist" Jul 21 13:47:00.684455 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.684426 2569 scope.go:117] "RemoveContainer" containerID="ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0" Jul 21 13:47:00.684704 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:00.684688 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0\": container with ID starting with ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0 not found: ID does not exist" containerID="ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0" Jul 21 13:47:00.684753 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.684707 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0"} err="failed to get container status \"ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0\": rpc error: code = NotFound desc = could not find container \"ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0\": container with ID starting with ba94ee3b5ff04a318f7eca0dee00c62e2c8cdcdfa3a02621d54b65b15834f1c0 not found: ID does not exist" Jul 21 13:47:00.684753 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.684720 2569 scope.go:117] "RemoveContainer" containerID="9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d" Jul 21 13:47:00.684927 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:00.684913 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d\": container with ID starting with 9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d not found: ID does not exist" containerID="9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d" Jul 21 13:47:00.684967 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.684929 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d"} err="failed to get container status \"9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d\": rpc error: code = NotFound desc = could not find container \"9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d\": container with ID starting with 9024873c8070d2f372eb445ef385b4128e5cb018f52404d2f53783038ab1064d not found: ID does not exist" Jul 21 13:47:00.684967 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.684940 2569 scope.go:117] "RemoveContainer" containerID="8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0" Jul 21 13:47:00.685148 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:00.685126 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0\": container with ID starting with 8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0 not found: ID does not exist" containerID="8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0" Jul 21 13:47:00.685234 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.685147 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0"} err="failed to get container status \"8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0\": rpc error: code = NotFound desc = could not find container \"8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0\": container with ID starting with 8b0e7f1f5abd556e7534350b1a36f965fd2e004f9ef93f1779ac29c3d0f139a0 not found: ID does not exist" Jul 21 13:47:00.685234 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.685159 2569 scope.go:117] "RemoveContainer" containerID="4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae" Jul 21 13:47:00.685347 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:00.685331 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae\": container with ID starting with 4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae not found: ID does not exist" containerID="4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae" Jul 21 13:47:00.685386 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.685349 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae"} err="failed to get container status \"4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae\": rpc error: code = NotFound desc = could not find container \"4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae\": container with ID starting with 4e46fdeddbec4eda7306eba263633544770f9e95b8e488429b0b8c2b77fac0ae not found: ID does not exist" Jul 21 13:47:00.748229 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.748195 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:47:00.769428 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.769398 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:47:00.794464 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.794437 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31cfef40-9134-4b59-887c-097dadf6a797" path="/var/lib/kubelet/pods/31cfef40-9134-4b59-887c-097dadf6a797/volumes" Jul 21 13:47:00.836115 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836090 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:47:00.836407 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836396 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="alertmanager" Jul 21 13:47:00.836450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836409 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="alertmanager" Jul 21 13:47:00.836450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836417 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="config-reloader" Jul 21 13:47:00.836450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836422 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="config-reloader" Jul 21 13:47:00.836450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836432 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4790404b-c94b-40f6-9ffc-932b6798c742" containerName="console" Jul 21 13:47:00.836450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836437 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4790404b-c94b-40f6-9ffc-932b6798c742" containerName="console" Jul 21 13:47:00.836450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836445 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy-web" Jul 21 13:47:00.836450 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836450 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy-web" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836459 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23e7245c-7db9-45c8-af26-00e216c4bfe0" containerName="console" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836464 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23e7245c-7db9-45c8-af26-00e216c4bfe0" containerName="console" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836471 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58be5d89-df08-45b5-8fed-5afc5a84f585" containerName="console" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836476 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="58be5d89-df08-45b5-8fed-5afc5a84f585" containerName="console" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836489 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="init-config-reloader" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836494 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="init-config-reloader" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836501 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836507 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836514 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy-metric" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836519 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy-metric" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836525 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="prom-label-proxy" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836530 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="prom-label-proxy" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836589 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="58be5d89-df08-45b5-8fed-5afc5a84f585" containerName="console" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836596 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4790404b-c94b-40f6-9ffc-932b6798c742" containerName="console" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836601 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy-metric" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836609 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="23e7245c-7db9-45c8-af26-00e216c4bfe0" containerName="console" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836614 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="config-reloader" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836619 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836625 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="alertmanager" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836631 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="kube-rbac-proxy-web" Jul 21 13:47:00.836657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.836637 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cfef40-9134-4b59-887c-097dadf6a797" containerName="prom-label-proxy" Jul 21 13:47:00.841868 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.841851 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.848527 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.848509 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 21 13:47:00.848638 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.848578 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 21 13:47:00.848726 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.848707 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 21 13:47:00.848779 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.848757 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 21 13:47:00.848831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.848718 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 21 13:47:00.848831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.848781 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 21 13:47:00.848831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.848707 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 21 13:47:00.849157 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.849141 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 21 13:47:00.849434 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.849419 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dskmg\"" Jul 21 13:47:00.855689 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.855666 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 21 13:47:00.863467 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.863446 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:47:00.897939 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.897881 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.897939 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.897908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898053 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.897947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898053 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nvb\" (UniqueName: \"kubernetes.io/projected/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-kube-api-access-r9nvb\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898053 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898050 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898152 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898152 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898152 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898127 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-web-config\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898173 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-config-out\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.898401 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.898270 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999025 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.998995 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nvb\" (UniqueName: \"kubernetes.io/projected/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-kube-api-access-r9nvb\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-web-config\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-config-out\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999162 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999499 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999499 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999716 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999692 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999798 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999753 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999888 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999830 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999888 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999856 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:00.999992 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:00.999897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.001529 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.001503 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.001642 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.001604 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.002266 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.002086 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.002266 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.002222 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-web-config\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.002406 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.002268 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.002406 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.002303 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.002741 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.002675 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.002903 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.002876 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-config-out\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.003296 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.003278 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.003389 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.003283 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.003711 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.003694 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.007325 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.007308 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nvb\" (UniqueName: \"kubernetes.io/projected/6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69-kube-api-access-r9nvb\") pod \"alertmanager-main-0\" (UID: \"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.151575 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.151461 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 13:47:01.293765 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.293732 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 13:47:01.297283 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:47:01.297247 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0e8ee1_a2df_4516_bf4f_f0d08c6fed69.slice/crio-45f73d872909a194a2489c62d965b6cd0f93ac43064761e5518b94b6f376200b WatchSource:0}: Error finding container 45f73d872909a194a2489c62d965b6cd0f93ac43064761e5518b94b6f376200b: Status 404 returned error can't find the container with id 45f73d872909a194a2489c62d965b6cd0f93ac43064761e5518b94b6f376200b Jul 21 13:47:01.643424 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.643386 2569 generic.go:356] "Generic (PLEG): container finished" podID="6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69" containerID="290682f4c647fd899bbeb2920040518fbbb9c88f6135233930818fbbbf1ec56b" exitCode=0 Jul 21 13:47:01.643600 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.643431 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69","Type":"ContainerDied","Data":"290682f4c647fd899bbeb2920040518fbbb9c88f6135233930818fbbbf1ec56b"} Jul 21 13:47:01.643600 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:01.643509 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69","Type":"ContainerStarted","Data":"45f73d872909a194a2489c62d965b6cd0f93ac43064761e5518b94b6f376200b"} Jul 21 13:47:02.650961 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:02.650926 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69","Type":"ContainerStarted","Data":"b8861e707de7dc5d1fd4a6b52426e11648863001f9349f300490cbb1a7d7f950"} Jul 21 13:47:02.651365 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:02.650991 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69","Type":"ContainerStarted","Data":"b842ebba539a5c89de80b7079766594b4b0c5cd5ed7da7d23b4df706c132f362"} Jul 21 13:47:02.651365 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:02.651030 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69","Type":"ContainerStarted","Data":"282d65b8e54a75a9074faea18e5e887860823b6d2cd3e544e2c0a8bb57086033"} Jul 21 13:47:02.651365 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:02.651055 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69","Type":"ContainerStarted","Data":"1d7550133eb04c6054720b84df5247ce700f5e061378d294cab3020c6a232048"} Jul 21 13:47:02.651365 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:02.651079 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69","Type":"ContainerStarted","Data":"6b27a89711a52fbfb26749994e2c0604dd946c7df2083b43a5c52e0116b4c6a9"} Jul 21 13:47:02.651365 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:02.651102 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69","Type":"ContainerStarted","Data":"54bc17f32b1a14c9e9ca1abf9e64ceed18397f29f35aff58df8d0b1aa4fc6f98"} Jul 21 13:47:02.681220 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:02.681160 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.666217097 podStartE2EDuration="2.681142742s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.925645ms" isStatefulPod=false podCreationTimestamp="2026-07-21 13:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:47:02.67907654 +0000 UTC m=+155.717146461" watchObservedRunningTime="2026-07-21 13:47:02.681142742 +0000 UTC m=+155.719212701" Jul 21 13:47:03.252729 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.252694 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:47:03.253033 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.252981 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="prometheus" containerID="cri-o://d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" gracePeriod=600 Jul 21 13:47:03.253109 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.253035 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="config-reloader" containerID="cri-o://491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" gracePeriod=600 Jul 21 13:47:03.253109 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.253044 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy-thanos" containerID="cri-o://746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" gracePeriod=600 Jul 21 13:47:03.253109 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.253063 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="thanos-sidecar" containerID="cri-o://26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" gracePeriod=600 Jul 21 13:47:03.253308 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.253004 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy" containerID="cri-o://8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" gracePeriod=600 Jul 21 13:47:03.253308 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.253154 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy-web" containerID="cri-o://e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" gracePeriod=600 Jul 21 13:47:03.515490 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.515422 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.625247 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625209 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-config\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625247 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625251 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-kube-rbac-proxy\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625498 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625275 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-trusted-ca-bundle\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625498 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625300 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625498 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625381 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-2\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625498 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625425 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-0\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625498 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625458 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-db\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625498 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625493 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-tls-assets\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625524 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-metrics-client-certs\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625582 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-config-out\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625627 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-thanos-prometheus-http-client-file\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625663 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-serving-certs-ca-bundle\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625703 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625749 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-kubelet-serving-ca-bundle\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625755 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625777 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-metrics-client-ca\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.625831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625831 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-tls\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.626262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625862 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h5rq\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-kube-api-access-8h5rq\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.626262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625893 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-grpc-tls\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.626262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625919 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-web-config\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.626262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.625965 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.626262 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.626002 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-1\") pod \"b1910624-6af9-4f6b-8841-795aeffa1c25\" (UID: \"b1910624-6af9-4f6b-8841-795aeffa1c25\") " Jul 21 13:47:03.626523 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.626298 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.626523 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.626326 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.626638 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.626611 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:03.626985 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.626679 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:03.627796 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.627760 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:03.628088 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.628059 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.628175 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.628114 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:03.628359 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.628333 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.628483 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.628456 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.628648 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.628626 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-config" (OuterVolumeSpecName: "config") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.628810 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.628786 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:47:03.629050 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.629023 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:47:03.630047 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.630014 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.630254 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.630207 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.630328 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.630290 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-kube-api-access-8h5rq" (OuterVolumeSpecName: "kube-api-access-8h5rq") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "kube-api-access-8h5rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:47:03.630585 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.630561 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.630656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.630639 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.630884 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.630863 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-config-out" (OuterVolumeSpecName: "config-out") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:47:03.631021 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.631000 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:47:03.640636 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.640613 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-web-config" (OuterVolumeSpecName: "web-config") pod "b1910624-6af9-4f6b-8841-795aeffa1c25" (UID: "b1910624-6af9-4f6b-8841-795aeffa1c25"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:47:03.658953 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.658930 2569 generic.go:356] "Generic (PLEG): container finished" podID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" exitCode=0 Jul 21 13:47:03.658953 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.658953 2569 generic.go:356] "Generic (PLEG): container finished" podID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" exitCode=0 Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.658965 2569 generic.go:356] "Generic (PLEG): container finished" podID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" exitCode=0 Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.658973 2569 generic.go:356] "Generic (PLEG): container finished" podID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" exitCode=0 Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.658980 2569 generic.go:356] "Generic (PLEG): container finished" podID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" exitCode=0 Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.658987 2569 generic.go:356] "Generic (PLEG): container finished" podID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" exitCode=0 Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659003 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659005 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerDied","Data":"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb"} Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerDied","Data":"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f"} Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659091 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerDied","Data":"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f"} Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerDied","Data":"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027"} Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659141 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerDied","Data":"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391"} Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659160 2569 scope.go:117] "RemoveContainer" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659165 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerDied","Data":"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b"} Jul 21 13:47:03.659332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.659189 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1910624-6af9-4f6b-8841-795aeffa1c25","Type":"ContainerDied","Data":"bf4fbd8ec81e3d267e69451fc6d38286eb54dec6eb3bbac9f40ff3d746ea64f1"} Jul 21 13:47:03.666966 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.666945 2569 scope.go:117] "RemoveContainer" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" Jul 21 13:47:03.674564 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.674521 2569 scope.go:117] "RemoveContainer" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" Jul 21 13:47:03.681147 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.681121 2569 scope.go:117] "RemoveContainer" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" Jul 21 13:47:03.685961 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.685939 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:47:03.692279 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.692258 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:47:03.692766 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.692750 2569 scope.go:117] "RemoveContainer" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" Jul 21 13:47:03.699237 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.699215 2569 scope.go:117] "RemoveContainer" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" Jul 21 13:47:03.706272 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.706257 2569 scope.go:117] "RemoveContainer" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" Jul 21 13:47:03.712863 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.712847 2569 scope.go:117] "RemoveContainer" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" Jul 21 13:47:03.713107 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:03.713088 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": container with ID starting with 746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb not found: ID does not exist" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" Jul 21 13:47:03.713158 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.713114 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb"} err="failed to get container status \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": rpc error: code = NotFound desc = could not find container \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": container with ID starting with 746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb not found: ID does not exist" Jul 21 13:47:03.713158 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.713132 2569 scope.go:117] "RemoveContainer" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" Jul 21 13:47:03.713356 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:03.713341 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": container with ID starting with 8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f not found: ID does not exist" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" Jul 21 13:47:03.713396 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.713358 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f"} err="failed to get container status \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": rpc error: code = NotFound desc = could not find container \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": container with ID starting with 8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f not found: ID does not exist" Jul 21 13:47:03.713396 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.713371 2569 scope.go:117] "RemoveContainer" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" Jul 21 13:47:03.713645 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:03.713628 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": container with ID starting with e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f not found: ID does not exist" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" Jul 21 13:47:03.713705 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.713650 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f"} err="failed to get container status \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": rpc error: code = NotFound desc = could not find container \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": container with ID starting with e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f not found: ID does not exist" Jul 21 13:47:03.713705 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.713666 2569 scope.go:117] "RemoveContainer" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" Jul 21 13:47:03.713883 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:03.713866 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": container with ID starting with 26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027 not found: ID does not exist" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" Jul 21 13:47:03.713921 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.713886 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027"} err="failed to get container status \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": rpc error: code = NotFound desc = could not find container \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": container with ID starting with 26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027 not found: ID does not exist" Jul 21 13:47:03.713921 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.713899 2569 scope.go:117] "RemoveContainer" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" Jul 21 13:47:03.714067 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:03.714054 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": container with ID starting with 491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391 not found: ID does not exist" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" Jul 21 13:47:03.714104 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714070 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391"} err="failed to get container status \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": rpc error: code = NotFound desc = could not find container \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": container with ID starting with 491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391 not found: ID does not exist" Jul 21 13:47:03.714104 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714081 2569 scope.go:117] "RemoveContainer" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" Jul 21 13:47:03.714259 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:03.714245 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": container with ID starting with d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b not found: ID does not exist" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" Jul 21 13:47:03.714298 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714261 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b"} err="failed to get container status \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": rpc error: code = NotFound desc = could not find container \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": container with ID starting with d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b not found: ID does not exist" Jul 21 13:47:03.714298 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714272 2569 scope.go:117] "RemoveContainer" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" Jul 21 13:47:03.714474 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:47:03.714461 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": container with ID starting with 749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b not found: ID does not exist" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" Jul 21 13:47:03.714508 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714476 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b"} err="failed to get container status \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": rpc error: code = NotFound desc = could not find container \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": container with ID starting with 749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b not found: ID does not exist" Jul 21 13:47:03.714508 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714487 2569 scope.go:117] "RemoveContainer" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" Jul 21 13:47:03.714706 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714689 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb"} err="failed to get container status \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": rpc error: code = NotFound desc = could not find container \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": container with ID starting with 746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb not found: ID does not exist" Jul 21 13:47:03.714750 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714707 2569 scope.go:117] "RemoveContainer" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" Jul 21 13:47:03.714911 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714890 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f"} err="failed to get container status \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": rpc error: code = NotFound desc = could not find container \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": container with ID starting with 8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f not found: ID does not exist" Jul 21 13:47:03.714960 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.714913 2569 scope.go:117] "RemoveContainer" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" Jul 21 13:47:03.715096 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715081 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f"} err="failed to get container status \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": rpc error: code = NotFound desc = could not find container \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": container with ID starting with e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f not found: ID does not exist" Jul 21 13:47:03.715138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715096 2569 scope.go:117] "RemoveContainer" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" Jul 21 13:47:03.715293 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715278 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027"} err="failed to get container status \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": rpc error: code = NotFound desc = could not find container \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": container with ID starting with 26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027 not found: ID does not exist" Jul 21 13:47:03.715293 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715293 2569 scope.go:117] "RemoveContainer" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" Jul 21 13:47:03.715466 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715452 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391"} err="failed to get container status \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": rpc error: code = NotFound desc = could not find container \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": container with ID starting with 491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391 not found: ID does not exist" Jul 21 13:47:03.715506 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715466 2569 scope.go:117] "RemoveContainer" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" Jul 21 13:47:03.715693 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715677 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b"} err="failed to get container status \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": rpc error: code = NotFound desc = could not find container \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": container with ID starting with d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b not found: ID does not exist" Jul 21 13:47:03.715693 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715692 2569 scope.go:117] "RemoveContainer" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" Jul 21 13:47:03.715868 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715853 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b"} err="failed to get container status \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": rpc error: code = NotFound desc = could not find container \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": container with ID starting with 749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b not found: ID does not exist" Jul 21 13:47:03.715911 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.715869 2569 scope.go:117] "RemoveContainer" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" Jul 21 13:47:03.716088 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.716071 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb"} err="failed to get container status \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": rpc error: code = NotFound desc = could not find container \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": container with ID starting with 746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb not found: ID does not exist" Jul 21 13:47:03.716131 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.716088 2569 scope.go:117] "RemoveContainer" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" Jul 21 13:47:03.716898 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.716860 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f"} err="failed to get container status \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": rpc error: code = NotFound desc = could not find container \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": container with ID starting with 8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f not found: ID does not exist" Jul 21 13:47:03.716898 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.716889 2569 scope.go:117] "RemoveContainer" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" Jul 21 13:47:03.717265 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.717242 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f"} err="failed to get container status \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": rpc error: code = NotFound desc = could not find container \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": container with ID starting with e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f not found: ID does not exist" Jul 21 13:47:03.717339 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.717266 2569 scope.go:117] "RemoveContainer" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" Jul 21 13:47:03.717594 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.717533 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027"} err="failed to get container status \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": rpc error: code = NotFound desc = could not find container \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": container with ID starting with 26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027 not found: ID does not exist" Jul 21 13:47:03.717695 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.717595 2569 scope.go:117] "RemoveContainer" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" Jul 21 13:47:03.717871 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.717853 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391"} err="failed to get container status \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": rpc error: code = NotFound desc = could not find container \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": container with ID starting with 491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391 not found: ID does not exist" Jul 21 13:47:03.717938 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.717871 2569 scope.go:117] "RemoveContainer" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" Jul 21 13:47:03.718095 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718076 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b"} err="failed to get container status \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": rpc error: code = NotFound desc = could not find container \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": container with ID starting with d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b not found: ID does not exist" Jul 21 13:47:03.718095 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718092 2569 scope.go:117] "RemoveContainer" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" Jul 21 13:47:03.718323 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718300 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b"} err="failed to get container status \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": rpc error: code = NotFound desc = could not find container \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": container with ID starting with 749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b not found: ID does not exist" Jul 21 13:47:03.718380 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718326 2569 scope.go:117] "RemoveContainer" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" Jul 21 13:47:03.718535 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718519 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb"} err="failed to get container status \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": rpc error: code = NotFound desc = could not find container \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": container with ID starting with 746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb not found: ID does not exist" Jul 21 13:47:03.718631 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718535 2569 scope.go:117] "RemoveContainer" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" Jul 21 13:47:03.718631 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718599 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:47:03.718761 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718743 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f"} err="failed to get container status \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": rpc error: code = NotFound desc = could not find container \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": container with ID starting with 8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f not found: ID does not exist" Jul 21 13:47:03.718800 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718769 2569 scope.go:117] "RemoveContainer" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" Jul 21 13:47:03.718996 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718982 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy" Jul 21 13:47:03.718996 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718980 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f"} err="failed to get container status \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": rpc error: code = NotFound desc = could not find container \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": container with ID starting with e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f not found: ID does not exist" Jul 21 13:47:03.719066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.718998 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy" Jul 21 13:47:03.719066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719003 2569 scope.go:117] "RemoveContainer" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" Jul 21 13:47:03.719066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719006 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy-thanos" Jul 21 13:47:03.719066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719047 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy-thanos" Jul 21 13:47:03.719066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719067 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="config-reloader" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719073 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="config-reloader" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719091 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy-web" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719100 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy-web" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719107 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="init-config-reloader" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719112 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="init-config-reloader" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719122 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="prometheus" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719128 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="prometheus" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719136 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="thanos-sidecar" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719141 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="thanos-sidecar" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719209 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy-web" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719219 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy-thanos" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719225 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="config-reloader" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719233 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="thanos-sidecar" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719239 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="kube-rbac-proxy" Jul 21 13:47:03.719253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719249 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" containerName="prometheus" Jul 21 13:47:03.719808 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719245 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027"} err="failed to get container status \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": rpc error: code = NotFound desc = could not find container \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": container with ID starting with 26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027 not found: ID does not exist" Jul 21 13:47:03.719808 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719318 2569 scope.go:117] "RemoveContainer" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" Jul 21 13:47:03.719808 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719572 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391"} err="failed to get container status \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": rpc error: code = NotFound desc = could not find container \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": container with ID starting with 491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391 not found: ID does not exist" Jul 21 13:47:03.719808 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719590 2569 scope.go:117] "RemoveContainer" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" Jul 21 13:47:03.719922 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719805 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b"} err="failed to get container status \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": rpc error: code = NotFound desc = could not find container \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": container with ID starting with d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b not found: ID does not exist" Jul 21 13:47:03.719922 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.719845 2569 scope.go:117] "RemoveContainer" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" Jul 21 13:47:03.720046 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720029 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b"} err="failed to get container status \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": rpc error: code = NotFound desc = could not find container \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": container with ID starting with 749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b not found: ID does not exist" Jul 21 13:47:03.720046 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720044 2569 scope.go:117] "RemoveContainer" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" Jul 21 13:47:03.720254 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720230 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb"} err="failed to get container status \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": rpc error: code = NotFound desc = could not find container \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": container with ID starting with 746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb not found: ID does not exist" Jul 21 13:47:03.720303 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720256 2569 scope.go:117] "RemoveContainer" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" Jul 21 13:47:03.720481 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720465 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f"} err="failed to get container status \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": rpc error: code = NotFound desc = could not find container \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": container with ID starting with 8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f not found: ID does not exist" Jul 21 13:47:03.720520 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720481 2569 scope.go:117] "RemoveContainer" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" Jul 21 13:47:03.720688 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720669 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f"} err="failed to get container status \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": rpc error: code = NotFound desc = could not find container \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": container with ID starting with e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f not found: ID does not exist" Jul 21 13:47:03.720729 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720689 2569 scope.go:117] "RemoveContainer" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" Jul 21 13:47:03.720869 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720851 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027"} err="failed to get container status \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": rpc error: code = NotFound desc = could not find container \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": container with ID starting with 26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027 not found: ID does not exist" Jul 21 13:47:03.720915 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.720869 2569 scope.go:117] "RemoveContainer" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" Jul 21 13:47:03.721185 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.721163 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391"} err="failed to get container status \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": rpc error: code = NotFound desc = could not find container \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": container with ID starting with 491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391 not found: ID does not exist" Jul 21 13:47:03.721185 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.721183 2569 scope.go:117] "RemoveContainer" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" Jul 21 13:47:03.721474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.721450 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b"} err="failed to get container status \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": rpc error: code = NotFound desc = could not find container \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": container with ID starting with d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b not found: ID does not exist" Jul 21 13:47:03.721474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.721473 2569 scope.go:117] "RemoveContainer" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" Jul 21 13:47:03.721784 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.721757 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b"} err="failed to get container status \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": rpc error: code = NotFound desc = could not find container \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": container with ID starting with 749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b not found: ID does not exist" Jul 21 13:47:03.721865 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.721785 2569 scope.go:117] "RemoveContainer" containerID="746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb" Jul 21 13:47:03.722062 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.722042 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb"} err="failed to get container status \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": rpc error: code = NotFound desc = could not find container \"746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb\": container with ID starting with 746bd2c8179d45498698900071e90d6edb648ecf796410ae7826631642841adb not found: ID does not exist" Jul 21 13:47:03.722062 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.722060 2569 scope.go:117] "RemoveContainer" containerID="8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f" Jul 21 13:47:03.722322 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.722302 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f"} err="failed to get container status \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": rpc error: code = NotFound desc = could not find container \"8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f\": container with ID starting with 8a841eee4bc026844baf7a0ddb4255150820129397c614036210d7b0ac35d79f not found: ID does not exist" Jul 21 13:47:03.722322 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.722322 2569 scope.go:117] "RemoveContainer" containerID="e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f" Jul 21 13:47:03.722643 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.722627 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f"} err="failed to get container status \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": rpc error: code = NotFound desc = could not find container \"e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f\": container with ID starting with e121bb2554506a9e969c22ab008cc6d8e5d3ca99985dc647361f25aebeb9f86f not found: ID does not exist" Jul 21 13:47:03.722643 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.722643 2569 scope.go:117] "RemoveContainer" containerID="26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027" Jul 21 13:47:03.722866 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.722848 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027"} err="failed to get container status \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": rpc error: code = NotFound desc = could not find container \"26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027\": container with ID starting with 26833b4d308b0e795360c96479825c2a26553c281e527d459b51030878869027 not found: ID does not exist" Jul 21 13:47:03.722913 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.722867 2569 scope.go:117] "RemoveContainer" containerID="491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391" Jul 21 13:47:03.723089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.723067 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391"} err="failed to get container status \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": rpc error: code = NotFound desc = could not find container \"491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391\": container with ID starting with 491fb18edfe87f85c2a4c4b9cf73af14c8b82eedb3861e0947d561b4243b9391 not found: ID does not exist" Jul 21 13:47:03.723089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.723088 2569 scope.go:117] "RemoveContainer" containerID="d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b" Jul 21 13:47:03.723330 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.723296 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b"} err="failed to get container status \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": rpc error: code = NotFound desc = could not find container \"d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b\": container with ID starting with d67af3326fa922fa0ce19147fb41a88476abcbd0f9d89d81a9691c33603ff07b not found: ID does not exist" Jul 21 13:47:03.723330 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.723318 2569 scope.go:117] "RemoveContainer" containerID="749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b" Jul 21 13:47:03.723566 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.723524 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b"} err="failed to get container status \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": rpc error: code = NotFound desc = could not find container \"749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b\": container with ID starting with 749985f030bfa458397090bdd70d6686080c6d3ca159a2b8aff6fd540abb4e1b not found: ID does not exist" Jul 21 13:47:03.724668 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.724654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.727142 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727121 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 13:47:03.727700 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727220 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 13:47:03.727700 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727222 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 13:47:03.727700 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727131 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 13:47:03.727700 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727475 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ftq8l5t3tu3rb\"" Jul 21 13:47:03.727700 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727644 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727761 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727858 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727878 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-metrics-client-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727893 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727908 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5rq\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-kube-api-access-8h5rq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727921 2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-grpc-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727934 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-web-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727947 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727965 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727976 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.727962 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728005 2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728027 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728034 2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-kube-rbac-proxy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728055 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728049 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728084 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728098 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-prometheus-k8s-db\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728112 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1910624-6af9-4f6b-8841-795aeffa1c25-tls-assets\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728124 2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-secret-metrics-client-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728136 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1910624-6af9-4f6b-8841-795aeffa1c25-config-out\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728162 2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1910624-6af9-4f6b-8841-795aeffa1c25-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728173 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1910624-6af9-4f6b-8841-795aeffa1c25-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728231 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728418 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728498 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 13:47:03.728762 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.728673 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vzq75\"" Jul 21 13:47:03.733943 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.733925 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 13:47:03.736832 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.736813 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 13:47:03.744131 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.743989 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:47:03.828673 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828579 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.828673 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828623 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.828673 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.828895 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.828895 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828706 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.828895 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828778 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.828895 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.828895 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.828895 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828913 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.828984 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.829030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psscs\" (UniqueName: \"kubernetes.io/projected/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-kube-api-access-psscs\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.829076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-config\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.829102 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829141 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.829137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829407 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.829165 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829407 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.829203 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.829407 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.829260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930131 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-config\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930131 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930361 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930163 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930361 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930361 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930361 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930407 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930654 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930661 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930803 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.930969 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.930948 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psscs\" (UniqueName: \"kubernetes.io/projected/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-kube-api-access-psscs\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.932425 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.932001 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.932767 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.932726 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.932767 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.932760 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.932925 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.932903 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.933657 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.933633 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.933752 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.933704 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.934159 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.934128 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.934492 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.934466 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.934584 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.934494 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-config\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.935299 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.935273 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.935393 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.935309 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.935464 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.935391 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.935524 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.935495 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.935524 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.935517 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.935646 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.935586 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.935840 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.935822 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.936815 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.936797 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:03.939923 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:03.939904 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psscs\" (UniqueName: \"kubernetes.io/projected/b1ca33b6-5d9c-4cda-84f6-0079874d8b03-kube-api-access-psscs\") pod \"prometheus-k8s-0\" (UID: \"b1ca33b6-5d9c-4cda-84f6-0079874d8b03\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:04.034527 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:04.034500 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:47:04.168386 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:04.168234 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 13:47:04.170463 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:47:04.170439 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ca33b6_5d9c_4cda_84f6_0079874d8b03.slice/crio-601d73a86901684e05d0f3dea4275515817ff249d6dd975ea32199a49a797324 WatchSource:0}: Error finding container 601d73a86901684e05d0f3dea4275515817ff249d6dd975ea32199a49a797324: Status 404 returned error can't find the container with id 601d73a86901684e05d0f3dea4275515817ff249d6dd975ea32199a49a797324 Jul 21 13:47:04.665778 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:04.665748 2569 generic.go:356] "Generic (PLEG): container finished" podID="b1ca33b6-5d9c-4cda-84f6-0079874d8b03" containerID="7d0714dcf99c7abc1bdf3826b8d16d980e30224e9068ee0947e6fc7aa8f61ec4" exitCode=0 Jul 21 13:47:04.666234 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:04.665836 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1ca33b6-5d9c-4cda-84f6-0079874d8b03","Type":"ContainerDied","Data":"7d0714dcf99c7abc1bdf3826b8d16d980e30224e9068ee0947e6fc7aa8f61ec4"} Jul 21 13:47:04.666234 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:04.665917 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1ca33b6-5d9c-4cda-84f6-0079874d8b03","Type":"ContainerStarted","Data":"601d73a86901684e05d0f3dea4275515817ff249d6dd975ea32199a49a797324"} Jul 21 13:47:04.803147 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:04.803119 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1910624-6af9-4f6b-8841-795aeffa1c25" path="/var/lib/kubelet/pods/b1910624-6af9-4f6b-8841-795aeffa1c25/volumes" Jul 21 13:47:05.674561 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:05.674504 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1ca33b6-5d9c-4cda-84f6-0079874d8b03","Type":"ContainerStarted","Data":"da46e556138d40d07051e873a7f27aad1e528cc97bdba52c87bfdbe599348f60"} Jul 21 13:47:05.674953 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:05.674598 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1ca33b6-5d9c-4cda-84f6-0079874d8b03","Type":"ContainerStarted","Data":"a422a8f48c3fcc8d5ea3a993e28db45f6d2835cda65687d61018ce47e46631aa"} Jul 21 13:47:05.674953 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:05.674628 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1ca33b6-5d9c-4cda-84f6-0079874d8b03","Type":"ContainerStarted","Data":"a2b83e7394fe55db76eea9a08a0df88015e382dee46c370ed521191ce966fe24"} Jul 21 13:47:05.674953 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:05.674653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1ca33b6-5d9c-4cda-84f6-0079874d8b03","Type":"ContainerStarted","Data":"937a4ffac4fa7b90c5645e7b5892e712675c0d555e06cf2782c3f612f708e06f"} Jul 21 13:47:05.674953 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:05.674676 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1ca33b6-5d9c-4cda-84f6-0079874d8b03","Type":"ContainerStarted","Data":"ff68298c707ead7ed1c577b71c4694ffc026eb6531e9f41cc8ef21319acea7fe"} Jul 21 13:47:05.674953 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:05.674698 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1ca33b6-5d9c-4cda-84f6-0079874d8b03","Type":"ContainerStarted","Data":"3a01643b06cc8b94954f4ee17be58c8f0de8efed8bffa3dd06741c3190ea5f1b"} Jul 21 13:47:05.705005 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:05.704951 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.687373181 podStartE2EDuration="2.704936054s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.562873ms" isStatefulPod=false podCreationTimestamp="2026-07-21 13:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:47:05.702155058 +0000 UTC m=+158.740225016" watchObservedRunningTime="2026-07-21 13:47:05.704936054 +0000 UTC m=+158.743005975" Jul 21 13:47:09.035080 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:47:09.035044 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:48:04.035432 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:04.035390 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:48:04.050921 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:04.050886 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:48:04.936589 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:04.936562 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 13:48:50.064696 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.064616 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h"] Jul 21 13:48:50.067829 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.067813 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" Jul 21 13:48:50.070420 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.070399 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 13:48:50.071391 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.071368 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sj498\"" Jul 21 13:48:50.071488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.071368 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 13:48:50.079339 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.079320 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h"] Jul 21 13:48:50.224326 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.224296 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/903bf617-543f-4316-ba08-af1c13d00524-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9zb4h\" (UID: \"903bf617-543f-4316-ba08-af1c13d00524\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" Jul 21 13:48:50.224487 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.224329 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdl4\" (UniqueName: \"kubernetes.io/projected/903bf617-543f-4316-ba08-af1c13d00524-kube-api-access-czdl4\") pod \"cert-manager-cainjector-559cdd8c87-9zb4h\" (UID: \"903bf617-543f-4316-ba08-af1c13d00524\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" Jul 21 13:48:50.325201 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.325128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/903bf617-543f-4316-ba08-af1c13d00524-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9zb4h\" (UID: \"903bf617-543f-4316-ba08-af1c13d00524\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" Jul 21 13:48:50.325201 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.325158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdl4\" (UniqueName: \"kubernetes.io/projected/903bf617-543f-4316-ba08-af1c13d00524-kube-api-access-czdl4\") pod \"cert-manager-cainjector-559cdd8c87-9zb4h\" (UID: \"903bf617-543f-4316-ba08-af1c13d00524\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" Jul 21 13:48:50.333334 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.333304 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/903bf617-543f-4316-ba08-af1c13d00524-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9zb4h\" (UID: \"903bf617-543f-4316-ba08-af1c13d00524\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" Jul 21 13:48:50.333490 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.333473 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdl4\" (UniqueName: \"kubernetes.io/projected/903bf617-543f-4316-ba08-af1c13d00524-kube-api-access-czdl4\") pod \"cert-manager-cainjector-559cdd8c87-9zb4h\" (UID: \"903bf617-543f-4316-ba08-af1c13d00524\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" Jul 21 13:48:50.391763 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.391739 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" Jul 21 13:48:50.514946 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:50.514919 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h"] Jul 21 13:48:50.519749 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:48:50.519723 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod903bf617_543f_4316_ba08_af1c13d00524.slice/crio-da3070d63a5c6685dc732b644ced3d00c375fdfabcb0c1c06d2716f59371e880 WatchSource:0}: Error finding container da3070d63a5c6685dc732b644ced3d00c375fdfabcb0c1c06d2716f59371e880: Status 404 returned error can't find the container with id da3070d63a5c6685dc732b644ced3d00c375fdfabcb0c1c06d2716f59371e880 Jul 21 13:48:51.108607 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:51.108570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" event={"ID":"903bf617-543f-4316-ba08-af1c13d00524","Type":"ContainerStarted","Data":"da3070d63a5c6685dc732b644ced3d00c375fdfabcb0c1c06d2716f59371e880"} Jul 21 13:48:54.125870 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:54.125829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" event={"ID":"903bf617-543f-4316-ba08-af1c13d00524","Type":"ContainerStarted","Data":"dae2f48453f6665b9324082823d1c73a17f04c08d732afffc46af9e8c8435179"} Jul 21 13:48:59.031712 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.031661 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9zb4h" podStartSLOduration=6.053006819 podStartE2EDuration="9.031644963s" totalImagesPullingTime="2.978638144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:48:54.140938214 +0000 UTC m=+267.179008148" watchObservedRunningTime="2026-07-21 13:48:59.031644963 +0000 UTC m=+272.069714892" Jul 21 13:48:59.032161 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.032126 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-48hcp"] Jul 21 13:48:59.035433 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.035413 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-48hcp" Jul 21 13:48:59.038097 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.038072 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nrwsw\"" Jul 21 13:48:59.047298 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.047278 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-48hcp"] Jul 21 13:48:59.199675 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.199646 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ctvb\" (UniqueName: \"kubernetes.io/projected/fce1194f-1567-4e59-9224-e805dd8c145c-kube-api-access-8ctvb\") pod \"cert-manager-55b4d954d5-48hcp\" (UID: \"fce1194f-1567-4e59-9224-e805dd8c145c\") " pod="cert-manager/cert-manager-55b4d954d5-48hcp" Jul 21 13:48:59.199837 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.199691 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fce1194f-1567-4e59-9224-e805dd8c145c-bound-sa-token\") pod \"cert-manager-55b4d954d5-48hcp\" (UID: \"fce1194f-1567-4e59-9224-e805dd8c145c\") " pod="cert-manager/cert-manager-55b4d954d5-48hcp" Jul 21 13:48:59.300499 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.300436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fce1194f-1567-4e59-9224-e805dd8c145c-bound-sa-token\") pod \"cert-manager-55b4d954d5-48hcp\" (UID: \"fce1194f-1567-4e59-9224-e805dd8c145c\") " pod="cert-manager/cert-manager-55b4d954d5-48hcp" Jul 21 13:48:59.300621 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.300598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ctvb\" (UniqueName: \"kubernetes.io/projected/fce1194f-1567-4e59-9224-e805dd8c145c-kube-api-access-8ctvb\") pod \"cert-manager-55b4d954d5-48hcp\" (UID: \"fce1194f-1567-4e59-9224-e805dd8c145c\") " pod="cert-manager/cert-manager-55b4d954d5-48hcp" Jul 21 13:48:59.309520 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.309493 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fce1194f-1567-4e59-9224-e805dd8c145c-bound-sa-token\") pod \"cert-manager-55b4d954d5-48hcp\" (UID: \"fce1194f-1567-4e59-9224-e805dd8c145c\") " pod="cert-manager/cert-manager-55b4d954d5-48hcp" Jul 21 13:48:59.309658 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.309640 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ctvb\" (UniqueName: \"kubernetes.io/projected/fce1194f-1567-4e59-9224-e805dd8c145c-kube-api-access-8ctvb\") pod \"cert-manager-55b4d954d5-48hcp\" (UID: \"fce1194f-1567-4e59-9224-e805dd8c145c\") " pod="cert-manager/cert-manager-55b4d954d5-48hcp" Jul 21 13:48:59.344580 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.344555 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-48hcp" Jul 21 13:48:59.472121 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:48:59.472028 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-48hcp"] Jul 21 13:48:59.474238 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:48:59.474206 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfce1194f_1567_4e59_9224_e805dd8c145c.slice/crio-3ce169e1a3fb2736fa79e0658ac99c1ca74dbc154a5d34adb4de32bc02a70ff4 WatchSource:0}: Error finding container 3ce169e1a3fb2736fa79e0658ac99c1ca74dbc154a5d34adb4de32bc02a70ff4: Status 404 returned error can't find the container with id 3ce169e1a3fb2736fa79e0658ac99c1ca74dbc154a5d34adb4de32bc02a70ff4 Jul 21 13:49:00.151014 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:00.150980 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-48hcp" event={"ID":"fce1194f-1567-4e59-9224-e805dd8c145c","Type":"ContainerStarted","Data":"f3b82c81bc1be7622915d1bc67fea6e50d9c10532957db07df07d3c080dface7"} Jul 21 13:49:00.151385 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:00.151022 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-48hcp" event={"ID":"fce1194f-1567-4e59-9224-e805dd8c145c","Type":"ContainerStarted","Data":"3ce169e1a3fb2736fa79e0658ac99c1ca74dbc154a5d34adb4de32bc02a70ff4"} Jul 21 13:49:00.168184 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:00.168131 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-48hcp" podStartSLOduration=1.168116768 podStartE2EDuration="1.168116768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:49:00.167943683 +0000 UTC m=+273.206013603" watchObservedRunningTime="2026-07-21 13:49:00.168116768 +0000 UTC m=+273.206186689" Jul 21 13:49:19.128472 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.128433 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z"] Jul 21 13:49:19.131685 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.131666 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.135307 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.135284 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 13:49:19.135852 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.135833 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 13:49:19.135921 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.135848 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 13:49:19.136272 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.136258 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 13:49:19.136331 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.136305 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-h5js4\"" Jul 21 13:49:19.136331 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.136318 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 13:49:19.148001 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.147978 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z"] Jul 21 13:49:19.167640 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.167616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/818687d7-7030-4ae3-bc89-36a5d6713319-manager-config\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.167763 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.167675 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/818687d7-7030-4ae3-bc89-36a5d6713319-cert\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.167763 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.167732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/818687d7-7030-4ae3-bc89-36a5d6713319-metrics-cert\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.167841 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.167760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqqz\" (UniqueName: \"kubernetes.io/projected/818687d7-7030-4ae3-bc89-36a5d6713319-kube-api-access-tfqqz\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.268896 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.268860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/818687d7-7030-4ae3-bc89-36a5d6713319-manager-config\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.269082 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.268924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/818687d7-7030-4ae3-bc89-36a5d6713319-cert\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.269082 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.268946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/818687d7-7030-4ae3-bc89-36a5d6713319-metrics-cert\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.269082 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.268963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqqz\" (UniqueName: \"kubernetes.io/projected/818687d7-7030-4ae3-bc89-36a5d6713319-kube-api-access-tfqqz\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.269623 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.269597 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/818687d7-7030-4ae3-bc89-36a5d6713319-manager-config\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.271400 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.271377 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/818687d7-7030-4ae3-bc89-36a5d6713319-metrics-cert\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.271518 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.271442 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/818687d7-7030-4ae3-bc89-36a5d6713319-cert\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.278509 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.278485 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqqz\" (UniqueName: \"kubernetes.io/projected/818687d7-7030-4ae3-bc89-36a5d6713319-kube-api-access-tfqqz\") pod \"lws-controller-manager-686f4b4fc5-vcn4z\" (UID: \"818687d7-7030-4ae3-bc89-36a5d6713319\") " pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.442321 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.442229 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:19.584998 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:19.584976 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z"] Jul 21 13:49:19.587581 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:49:19.587521 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818687d7_7030_4ae3_bc89_36a5d6713319.slice/crio-df42eb221b99d7e7c03593bcc9d42cbc5a49a39a660a0fae039d2281eb13d083 WatchSource:0}: Error finding container df42eb221b99d7e7c03593bcc9d42cbc5a49a39a660a0fae039d2281eb13d083: Status 404 returned error can't find the container with id df42eb221b99d7e7c03593bcc9d42cbc5a49a39a660a0fae039d2281eb13d083 Jul 21 13:49:20.236475 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:20.236434 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" event={"ID":"818687d7-7030-4ae3-bc89-36a5d6713319","Type":"ContainerStarted","Data":"df42eb221b99d7e7c03593bcc9d42cbc5a49a39a660a0fae039d2281eb13d083"} Jul 21 13:49:22.246858 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:22.246823 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" event={"ID":"818687d7-7030-4ae3-bc89-36a5d6713319","Type":"ContainerStarted","Data":"e22714d88932d47399c11996e08efc093f70634c93d64db0d0b771b58ffa1928"} Jul 21 13:49:22.247256 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:22.246955 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:25.803182 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.803126 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" podStartSLOduration=4.377292175 podStartE2EDuration="6.803107847s" totalImagesPullingTime="2.425815672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:49:22.27963353 +0000 UTC m=+295.317703467" watchObservedRunningTime="2026-07-21 13:49:25.803107847 +0000 UTC m=+298.841177785" Jul 21 13:49:25.803894 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.803874 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85"] Jul 21 13:49:25.807584 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.807563 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:25.810056 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.810036 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 13:49:25.810138 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.810064 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 13:49:25.810458 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.810440 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 13:49:25.810551 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.810523 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hwcdx\"" Jul 21 13:49:25.810625 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.810599 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 13:49:25.824119 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.824095 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85"] Jul 21 13:49:25.929166 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.929132 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb6bd71d-faad-4dda-8283-b2887028bbe5-tmp\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:25.929332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.929191 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb6bd71d-faad-4dda-8283-b2887028bbe5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:25.929332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.929234 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz4ps\" (UniqueName: \"kubernetes.io/projected/fb6bd71d-faad-4dda-8283-b2887028bbe5-kube-api-access-cz4ps\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:25.929332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.929267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb6bd71d-faad-4dda-8283-b2887028bbe5-manifests\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:25.929332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:25.929297 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb6bd71d-faad-4dda-8283-b2887028bbe5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.030244 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.030206 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb6bd71d-faad-4dda-8283-b2887028bbe5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.030444 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.030267 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4ps\" (UniqueName: \"kubernetes.io/projected/fb6bd71d-faad-4dda-8283-b2887028bbe5-kube-api-access-cz4ps\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.030444 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.030299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb6bd71d-faad-4dda-8283-b2887028bbe5-manifests\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.030444 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.030332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb6bd71d-faad-4dda-8283-b2887028bbe5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.030444 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.030378 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb6bd71d-faad-4dda-8283-b2887028bbe5-tmp\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.030800 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.030764 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb6bd71d-faad-4dda-8283-b2887028bbe5-manifests\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.032713 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.032690 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb6bd71d-faad-4dda-8283-b2887028bbe5-tmp\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.032912 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.032892 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb6bd71d-faad-4dda-8283-b2887028bbe5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.032974 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.032926 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb6bd71d-faad-4dda-8283-b2887028bbe5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.039848 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.039818 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4ps\" (UniqueName: \"kubernetes.io/projected/fb6bd71d-faad-4dda-8283-b2887028bbe5-kube-api-access-cz4ps\") pod \"opendatahub-operator-controller-manager-5d98d8798d-ccn85\" (UID: \"fb6bd71d-faad-4dda-8283-b2887028bbe5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.118768 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.118693 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:26.267852 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:49:26.267823 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6bd71d_faad_4dda_8283_b2887028bbe5.slice/crio-83db22f1b9ea2b91048e6a28e11a2c1d4ecab2a8167e55e83a38fbc83782225b WatchSource:0}: Error finding container 83db22f1b9ea2b91048e6a28e11a2c1d4ecab2a8167e55e83a38fbc83782225b: Status 404 returned error can't find the container with id 83db22f1b9ea2b91048e6a28e11a2c1d4ecab2a8167e55e83a38fbc83782225b Jul 21 13:49:26.271384 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:26.271361 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85"] Jul 21 13:49:27.271374 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:27.271276 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" event={"ID":"fb6bd71d-faad-4dda-8283-b2887028bbe5","Type":"ContainerStarted","Data":"83db22f1b9ea2b91048e6a28e11a2c1d4ecab2a8167e55e83a38fbc83782225b"} Jul 21 13:49:27.313757 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:27.313727 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 13:49:30.286418 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:30.286381 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" event={"ID":"fb6bd71d-faad-4dda-8283-b2887028bbe5","Type":"ContainerStarted","Data":"d4566bb53f8214a29292b4e7991b8876142567f6509225277c2eb6446cdd1eaa"} Jul 21 13:49:31.292617 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:31.292583 2569 generic.go:356] "Generic (PLEG): container finished" podID="fb6bd71d-faad-4dda-8283-b2887028bbe5" containerID="d4566bb53f8214a29292b4e7991b8876142567f6509225277c2eb6446cdd1eaa" exitCode=0 Jul 21 13:49:31.292977 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:31.292664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" event={"ID":"fb6bd71d-faad-4dda-8283-b2887028bbe5","Type":"ContainerDied","Data":"d4566bb53f8214a29292b4e7991b8876142567f6509225277c2eb6446cdd1eaa"} Jul 21 13:49:31.293111 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:31.293097 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 13:49:32.300499 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:32.300456 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" event={"ID":"fb6bd71d-faad-4dda-8283-b2887028bbe5","Type":"ContainerStarted","Data":"513818b945e84011c47ffd6d3907b6abe2c8dd5eae5c9abce7aa5ff79f171da5"} Jul 21 13:49:32.300891 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:32.300605 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:32.354913 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:32.354865 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" podStartSLOduration=2.764685724 podStartE2EDuration="7.354851078s" totalImagesPullingTime="3.507422805s" totalInitContainerRuntime="1.082742549s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:49:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:49:32.354467002 +0000 UTC m=+305.392536923" watchObservedRunningTime="2026-07-21 13:49:32.354851078 +0000 UTC m=+305.392920998" Jul 21 13:49:33.253900 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:33.253870 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-686f4b4fc5-vcn4z" Jul 21 13:49:43.307167 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:43.307136 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d98d8798d-ccn85" Jul 21 13:49:56.476056 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.476023 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54"] Jul 21 13:49:56.484842 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.484814 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:56.487331 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.487308 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 13:49:56.487331 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.487323 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 13:49:56.487486 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.487387 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 13:49:56.487486 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.487316 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 13:49:56.488493 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.488470 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g8lxn\"" Jul 21 13:49:56.493286 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.493263 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54"] Jul 21 13:49:56.602926 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.602887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4119d337-191a-4129-8626-ed2505a0854e-tls-certs\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:56.603102 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.602947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4119d337-191a-4129-8626-ed2505a0854e-tmp\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:56.603102 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.603035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpw86\" (UniqueName: \"kubernetes.io/projected/4119d337-191a-4129-8626-ed2505a0854e-kube-api-access-cpw86\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:56.704299 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.704257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4119d337-191a-4129-8626-ed2505a0854e-tls-certs\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:56.704495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.704319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4119d337-191a-4129-8626-ed2505a0854e-tmp\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:56.704495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.704362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpw86\" (UniqueName: \"kubernetes.io/projected/4119d337-191a-4129-8626-ed2505a0854e-kube-api-access-cpw86\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:56.704495 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:49:56.704421 2569 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 21 13:49:56.704495 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:49:56.704488 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4119d337-191a-4129-8626-ed2505a0854e-tls-certs podName:4119d337-191a-4129-8626-ed2505a0854e nodeName:}" failed. No retries permitted until 2026-07-21 13:49:57.204471453 +0000 UTC m=+330.242541352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4119d337-191a-4129-8626-ed2505a0854e-tls-certs") pod "kube-auth-proxy-55c6b9f88d-9hq54" (UID: "4119d337-191a-4129-8626-ed2505a0854e") : secret "kube-auth-proxy-tls" not found Jul 21 13:49:56.706780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.706759 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4119d337-191a-4129-8626-ed2505a0854e-tmp\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:56.722625 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:56.722595 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpw86\" (UniqueName: \"kubernetes.io/projected/4119d337-191a-4129-8626-ed2505a0854e-kube-api-access-cpw86\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:57.209949 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:57.209913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4119d337-191a-4129-8626-ed2505a0854e-tls-certs\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:57.212320 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:57.212288 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4119d337-191a-4129-8626-ed2505a0854e-tls-certs\") pod \"kube-auth-proxy-55c6b9f88d-9hq54\" (UID: \"4119d337-191a-4129-8626-ed2505a0854e\") " pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:57.396325 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:57.396274 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" Jul 21 13:49:57.551991 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:57.551963 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54"] Jul 21 13:49:57.554648 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:49:57.554619 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4119d337_191a_4129_8626_ed2505a0854e.slice/crio-5a82b8e433fc1f228bd82179c2844205f967bb61e3e3cc1a189bcaad377333bd WatchSource:0}: Error finding container 5a82b8e433fc1f228bd82179c2844205f967bb61e3e3cc1a189bcaad377333bd: Status 404 returned error can't find the container with id 5a82b8e433fc1f228bd82179c2844205f967bb61e3e3cc1a189bcaad377333bd Jul 21 13:49:58.425281 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:49:58.425245 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" event={"ID":"4119d337-191a-4129-8626-ed2505a0854e","Type":"ContainerStarted","Data":"5a82b8e433fc1f228bd82179c2844205f967bb61e3e3cc1a189bcaad377333bd"} Jul 21 13:50:01.440645 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:50:01.440604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" event={"ID":"4119d337-191a-4129-8626-ed2505a0854e","Type":"ContainerStarted","Data":"c15cef863ae0eec55440d6ac5481646d682f997d16b3003c2e4c401ca53b9358"} Jul 21 13:50:01.461913 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:50:01.461862 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55c6b9f88d-9hq54" podStartSLOduration=2.393120992 podStartE2EDuration="5.461846813s" totalImagesPullingTime="3.068725821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:50:01.461358594 +0000 UTC m=+334.499428515" watchObservedRunningTime="2026-07-21 13:50:01.461846813 +0000 UTC m=+334.499916734" Jul 21 13:51:40.389446 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.389413 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq"] Jul 21 13:51:40.392894 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.392875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.395446 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.395426 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 21 13:51:40.396391 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.396358 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 13:51:40.396495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.396417 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 21 13:51:40.396495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.396426 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 13:51:40.396495 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.396431 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sq6kq\"" Jul 21 13:51:40.403369 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.403346 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq"] Jul 21 13:51:40.419677 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.419655 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8js9\" (UniqueName: \"kubernetes.io/projected/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-kube-api-access-q8js9\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.419776 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.419746 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.419817 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.419794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.520968 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.520938 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.521088 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.520976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.521088 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.521036 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8js9\" (UniqueName: \"kubernetes.io/projected/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-kube-api-access-q8js9\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.521765 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.521747 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.523399 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.523375 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.530309 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.530289 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8js9\" (UniqueName: \"kubernetes.io/projected/3c4f19b8-8a85-45eb-bfaa-3df67c186c01-kube-api-access-q8js9\") pod \"kuadrant-console-plugin-66b5774985-fmkhq\" (UID: \"3c4f19b8-8a85-45eb-bfaa-3df67c186c01\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.701926 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.701855 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" Jul 21 13:51:40.830032 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.829969 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq"] Jul 21 13:51:40.832749 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:51:40.832720 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c4f19b8_8a85_45eb_bfaa_3df67c186c01.slice/crio-d0d3049acc14f0253ee3ee0c28fc0ffa5f2e2ab23275ad66c8007d11897f776c WatchSource:0}: Error finding container d0d3049acc14f0253ee3ee0c28fc0ffa5f2e2ab23275ad66c8007d11897f776c: Status 404 returned error can't find the container with id d0d3049acc14f0253ee3ee0c28fc0ffa5f2e2ab23275ad66c8007d11897f776c Jul 21 13:51:40.903492 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:40.903449 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" event={"ID":"3c4f19b8-8a85-45eb-bfaa-3df67c186c01","Type":"ContainerStarted","Data":"d0d3049acc14f0253ee3ee0c28fc0ffa5f2e2ab23275ad66c8007d11897f776c"} Jul 21 13:51:50.492839 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.492747 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2"] Jul 21 13:51:50.501391 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.501365 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:50.506253 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.506233 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-44l6z\"" Jul 21 13:51:50.515066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.515044 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2"] Jul 21 13:51:50.618929 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.618891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c47r2\" (UID: \"be622e5b-06d2-43d4-a534-48f0ccf9d9e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:50.619091 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.618989 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cchk9\" (UniqueName: \"kubernetes.io/projected/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-kube-api-access-cchk9\") pod \"kuadrant-operator-controller-manager-55467d9f7-c47r2\" (UID: \"be622e5b-06d2-43d4-a534-48f0ccf9d9e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:50.631975 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.631941 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2"] Jul 21 13:51:50.632256 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:51:50.632233 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-cchk9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" podUID="be622e5b-06d2-43d4-a534-48f0ccf9d9e7" Jul 21 13:51:50.717381 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.717344 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2"] Jul 21 13:51:50.720577 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.720529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c47r2\" (UID: \"be622e5b-06d2-43d4-a534-48f0ccf9d9e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:50.720722 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.720601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cchk9\" (UniqueName: \"kubernetes.io/projected/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-kube-api-access-cchk9\") pod \"kuadrant-operator-controller-manager-55467d9f7-c47r2\" (UID: \"be622e5b-06d2-43d4-a534-48f0ccf9d9e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:50.720951 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.720929 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c47r2\" (UID: \"be622e5b-06d2-43d4-a534-48f0ccf9d9e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:50.727878 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.727849 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2"] Jul 21 13:51:50.732177 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:51:50.732152 2569 projected.go:196] Error preparing data for projected volume kube-api-access-cchk9 for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2: failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-c47r2" not found Jul 21 13:51:50.732339 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:51:50.732226 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-kube-api-access-cchk9 podName:be622e5b-06d2-43d4-a534-48f0ccf9d9e7 nodeName:}" failed. No retries permitted until 2026-07-21 13:51:51.232205124 +0000 UTC m=+444.270275027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cchk9" (UniqueName: "kubernetes.io/projected/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-kube-api-access-cchk9") pod "kuadrant-operator-controller-manager-55467d9f7-c47r2" (UID: "be622e5b-06d2-43d4-a534-48f0ccf9d9e7") : failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-c47r2" not found Jul 21 13:51:50.832575 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.832461 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs"] Jul 21 13:51:50.854947 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.854915 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:51:50.860106 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.859914 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs"] Jul 21 13:51:50.922990 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.922957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwmx\" (UniqueName: \"kubernetes.io/projected/6439a955-47dd-4d23-a3a3-7a03d76706e1-kube-api-access-4fwmx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z69gs\" (UID: \"6439a955-47dd-4d23-a3a3-7a03d76706e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:51:50.923183 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.923003 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6439a955-47dd-4d23-a3a3-7a03d76706e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z69gs\" (UID: \"6439a955-47dd-4d23-a3a3-7a03d76706e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:51:50.961190 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.961154 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:50.974685 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:50.974654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:50.977202 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:51:50.977165 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c47r2\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="be622e5b-06d2-43d4-a534-48f0ccf9d9e7" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:51.024129 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.024092 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-extensions-socket-volume\") pod \"be622e5b-06d2-43d4-a534-48f0ccf9d9e7\" (UID: \"be622e5b-06d2-43d4-a534-48f0ccf9d9e7\") " Jul 21 13:51:51.024294 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.024255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwmx\" (UniqueName: \"kubernetes.io/projected/6439a955-47dd-4d23-a3a3-7a03d76706e1-kube-api-access-4fwmx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z69gs\" (UID: \"6439a955-47dd-4d23-a3a3-7a03d76706e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:51:51.024344 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.024296 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6439a955-47dd-4d23-a3a3-7a03d76706e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z69gs\" (UID: \"6439a955-47dd-4d23-a3a3-7a03d76706e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:51:51.024399 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.024382 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "be622e5b-06d2-43d4-a534-48f0ccf9d9e7" (UID: "be622e5b-06d2-43d4-a534-48f0ccf9d9e7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:51:51.024465 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.024445 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-extensions-socket-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:51:51.024507 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.024471 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cchk9\" (UniqueName: \"kubernetes.io/projected/be622e5b-06d2-43d4-a534-48f0ccf9d9e7-kube-api-access-cchk9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:51:51.024774 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.024752 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6439a955-47dd-4d23-a3a3-7a03d76706e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z69gs\" (UID: \"6439a955-47dd-4d23-a3a3-7a03d76706e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:51:51.037090 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.037066 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwmx\" (UniqueName: \"kubernetes.io/projected/6439a955-47dd-4d23-a3a3-7a03d76706e1-kube-api-access-4fwmx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z69gs\" (UID: \"6439a955-47dd-4d23-a3a3-7a03d76706e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:51:51.170247 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.170116 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:51:51.966308 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:51.966279 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:51.971869 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:51:51.971830 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c47r2\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="be622e5b-06d2-43d4-a534-48f0ccf9d9e7" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:51.976509 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:51:51.976483 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c47r2\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="be622e5b-06d2-43d4-a534-48f0ccf9d9e7" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c47r2" Jul 21 13:51:52.796931 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:51:52.796889 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be622e5b-06d2-43d4-a534-48f0ccf9d9e7" path="/var/lib/kubelet/pods/be622e5b-06d2-43d4-a534-48f0ccf9d9e7/volumes" Jul 21 13:52:01.524884 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:01.524857 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs"] Jul 21 13:52:01.526291 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:52:01.526263 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6439a955_47dd_4d23_a3a3_7a03d76706e1.slice/crio-86e77ed42453864dfbf36e06a00f1cd3dc0a43704692075bf12010258a465f8c WatchSource:0}: Error finding container 86e77ed42453864dfbf36e06a00f1cd3dc0a43704692075bf12010258a465f8c: Status 404 returned error can't find the container with id 86e77ed42453864dfbf36e06a00f1cd3dc0a43704692075bf12010258a465f8c Jul 21 13:52:02.020524 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:02.020488 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" event={"ID":"6439a955-47dd-4d23-a3a3-7a03d76706e1","Type":"ContainerStarted","Data":"86e77ed42453864dfbf36e06a00f1cd3dc0a43704692075bf12010258a465f8c"} Jul 21 13:52:02.021856 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:02.021832 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" event={"ID":"3c4f19b8-8a85-45eb-bfaa-3df67c186c01","Type":"ContainerStarted","Data":"61051ef5a8b26e652a4dbd42de4261ec53f4cd7f84a5b76b6c01e422ec678720"} Jul 21 13:52:02.046780 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:02.046734 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fmkhq" podStartSLOduration=1.416069963 podStartE2EDuration="22.046719432s" totalImagesPullingTime="20.630649469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:52:02.043600209 +0000 UTC m=+455.081670127" watchObservedRunningTime="2026-07-21 13:52:02.046719432 +0000 UTC m=+455.084789354" Jul 21 13:52:03.108281 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:03.108248 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs"] Jul 21 13:52:08.053477 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.053443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" event={"ID":"6439a955-47dd-4d23-a3a3-7a03d76706e1","Type":"ContainerStarted","Data":"f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf"} Jul 21 13:52:08.053957 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.053508 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" podUID="6439a955-47dd-4d23-a3a3-7a03d76706e1" containerName="manager" containerID="cri-o://f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf" gracePeriod=10 Jul 21 13:52:08.053957 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.053655 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:52:08.079068 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.079008 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" podStartSLOduration=12.628667262 podStartE2EDuration="18.078993423s" totalImagesPullingTime="5.450326161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:52:08.078427522 +0000 UTC m=+461.116497444" watchObservedRunningTime="2026-07-21 13:52:08.078993423 +0000 UTC m=+461.117063346" Jul 21 13:52:08.347771 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.347737 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:52:08.495059 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.495025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6439a955-47dd-4d23-a3a3-7a03d76706e1-extensions-socket-volume\") pod \"6439a955-47dd-4d23-a3a3-7a03d76706e1\" (UID: \"6439a955-47dd-4d23-a3a3-7a03d76706e1\") " Jul 21 13:52:08.495243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.495098 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fwmx\" (UniqueName: \"kubernetes.io/projected/6439a955-47dd-4d23-a3a3-7a03d76706e1-kube-api-access-4fwmx\") pod \"6439a955-47dd-4d23-a3a3-7a03d76706e1\" (UID: \"6439a955-47dd-4d23-a3a3-7a03d76706e1\") " Jul 21 13:52:08.495487 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.495462 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6439a955-47dd-4d23-a3a3-7a03d76706e1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6439a955-47dd-4d23-a3a3-7a03d76706e1" (UID: "6439a955-47dd-4d23-a3a3-7a03d76706e1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:52:08.497298 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.497266 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6439a955-47dd-4d23-a3a3-7a03d76706e1-kube-api-access-4fwmx" (OuterVolumeSpecName: "kube-api-access-4fwmx") pod "6439a955-47dd-4d23-a3a3-7a03d76706e1" (UID: "6439a955-47dd-4d23-a3a3-7a03d76706e1"). InnerVolumeSpecName "kube-api-access-4fwmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:52:08.596226 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.596156 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwmx\" (UniqueName: \"kubernetes.io/projected/6439a955-47dd-4d23-a3a3-7a03d76706e1-kube-api-access-4fwmx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:52:08.596226 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:08.596183 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6439a955-47dd-4d23-a3a3-7a03d76706e1-extensions-socket-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:52:09.059556 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.059515 2569 generic.go:356] "Generic (PLEG): container finished" podID="6439a955-47dd-4d23-a3a3-7a03d76706e1" containerID="f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf" exitCode=0 Jul 21 13:52:09.059945 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.059572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" event={"ID":"6439a955-47dd-4d23-a3a3-7a03d76706e1","Type":"ContainerDied","Data":"f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf"} Jul 21 13:52:09.059945 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.059633 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" event={"ID":"6439a955-47dd-4d23-a3a3-7a03d76706e1","Type":"ContainerDied","Data":"86e77ed42453864dfbf36e06a00f1cd3dc0a43704692075bf12010258a465f8c"} Jul 21 13:52:09.059945 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.059577 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs" Jul 21 13:52:09.059945 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.059660 2569 scope.go:117] "RemoveContainer" containerID="f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf" Jul 21 13:52:09.067767 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.067748 2569 scope.go:117] "RemoveContainer" containerID="f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf" Jul 21 13:52:09.068016 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:52:09.067997 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf\": container with ID starting with f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf not found: ID does not exist" containerID="f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf" Jul 21 13:52:09.068072 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.068024 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf"} err="failed to get container status \"f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf\": rpc error: code = NotFound desc = could not find container \"f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf\": container with ID starting with f3407e80f8159300a672148e163600b63fc3497a31b2a0af7f8e9c0a14887cbf not found: ID does not exist" Jul 21 13:52:09.088175 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.088151 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs"] Jul 21 13:52:09.095255 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:09.095232 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z69gs"] Jul 21 13:52:10.795951 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:10.795919 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6439a955-47dd-4d23-a3a3-7a03d76706e1" path="/var/lib/kubelet/pods/6439a955-47dd-4d23-a3a3-7a03d76706e1/volumes" Jul 21 13:52:21.316971 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.316929 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cmvrj"] Jul 21 13:52:21.317359 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.317283 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6439a955-47dd-4d23-a3a3-7a03d76706e1" containerName="manager" Jul 21 13:52:21.317359 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.317294 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6439a955-47dd-4d23-a3a3-7a03d76706e1" containerName="manager" Jul 21 13:52:21.317433 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.317372 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6439a955-47dd-4d23-a3a3-7a03d76706e1" containerName="manager" Jul 21 13:52:21.320289 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.320269 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:21.323144 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.323118 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 13:52:21.343013 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.342986 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cmvrj"] Jul 21 13:52:21.393759 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.393721 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cmvrj"] Jul 21 13:52:21.408245 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.408205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjmj\" (UniqueName: \"kubernetes.io/projected/04045f92-3e45-49e1-b160-44d9435fe0de-kube-api-access-2fjmj\") pod \"limitador-limitador-64c94fd64f-cmvrj\" (UID: \"04045f92-3e45-49e1-b160-44d9435fe0de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:21.408376 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.408260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04045f92-3e45-49e1-b160-44d9435fe0de-config-file\") pod \"limitador-limitador-64c94fd64f-cmvrj\" (UID: \"04045f92-3e45-49e1-b160-44d9435fe0de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:21.509948 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.509907 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjmj\" (UniqueName: \"kubernetes.io/projected/04045f92-3e45-49e1-b160-44d9435fe0de-kube-api-access-2fjmj\") pod \"limitador-limitador-64c94fd64f-cmvrj\" (UID: \"04045f92-3e45-49e1-b160-44d9435fe0de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:21.510239 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.510219 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04045f92-3e45-49e1-b160-44d9435fe0de-config-file\") pod \"limitador-limitador-64c94fd64f-cmvrj\" (UID: \"04045f92-3e45-49e1-b160-44d9435fe0de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:21.511213 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.511189 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04045f92-3e45-49e1-b160-44d9435fe0de-config-file\") pod \"limitador-limitador-64c94fd64f-cmvrj\" (UID: \"04045f92-3e45-49e1-b160-44d9435fe0de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:21.529316 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.529288 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjmj\" (UniqueName: \"kubernetes.io/projected/04045f92-3e45-49e1-b160-44d9435fe0de-kube-api-access-2fjmj\") pod \"limitador-limitador-64c94fd64f-cmvrj\" (UID: \"04045f92-3e45-49e1-b160-44d9435fe0de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:21.631494 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.631449 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:21.769051 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:21.768976 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cmvrj"] Jul 21 13:52:21.771275 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:52:21.771252 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04045f92_3e45_49e1_b160_44d9435fe0de.slice/crio-b0d3a09ea1669851f1f486fbcf5156ce7c30e1e68a3c3f2a6e89290b6777f2a2 WatchSource:0}: Error finding container b0d3a09ea1669851f1f486fbcf5156ce7c30e1e68a3c3f2a6e89290b6777f2a2: Status 404 returned error can't find the container with id b0d3a09ea1669851f1f486fbcf5156ce7c30e1e68a3c3f2a6e89290b6777f2a2 Jul 21 13:52:22.125030 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:22.124950 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" event={"ID":"04045f92-3e45-49e1-b160-44d9435fe0de","Type":"ContainerStarted","Data":"b0d3a09ea1669851f1f486fbcf5156ce7c30e1e68a3c3f2a6e89290b6777f2a2"} Jul 21 13:52:25.141763 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:25.141678 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" event={"ID":"04045f92-3e45-49e1-b160-44d9435fe0de","Type":"ContainerStarted","Data":"1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b"} Jul 21 13:52:25.142114 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:25.141795 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:25.160473 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:25.160427 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" podStartSLOduration=1.137366103 podStartE2EDuration="4.160413876s" totalImagesPullingTime="3.023047773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:52:25.158333173 +0000 UTC m=+478.196403096" watchObservedRunningTime="2026-07-21 13:52:25.160413876 +0000 UTC m=+478.198483797" Jul 21 13:52:36.147614 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:36.147579 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:36.409781 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:36.409705 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cmvrj"] Jul 21 13:52:36.409915 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:36.409896 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" podUID="04045f92-3e45-49e1-b160-44d9435fe0de" containerName="limitador" containerID="cri-o://1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b" gracePeriod=30 Jul 21 13:52:36.982723 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:36.982701 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:37.056553 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.056518 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fjmj\" (UniqueName: \"kubernetes.io/projected/04045f92-3e45-49e1-b160-44d9435fe0de-kube-api-access-2fjmj\") pod \"04045f92-3e45-49e1-b160-44d9435fe0de\" (UID: \"04045f92-3e45-49e1-b160-44d9435fe0de\") " Jul 21 13:52:37.056730 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.056587 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04045f92-3e45-49e1-b160-44d9435fe0de-config-file\") pod \"04045f92-3e45-49e1-b160-44d9435fe0de\" (UID: \"04045f92-3e45-49e1-b160-44d9435fe0de\") " Jul 21 13:52:37.056986 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.056955 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04045f92-3e45-49e1-b160-44d9435fe0de-config-file" (OuterVolumeSpecName: "config-file") pod "04045f92-3e45-49e1-b160-44d9435fe0de" (UID: "04045f92-3e45-49e1-b160-44d9435fe0de"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:52:37.058551 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.058519 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04045f92-3e45-49e1-b160-44d9435fe0de-kube-api-access-2fjmj" (OuterVolumeSpecName: "kube-api-access-2fjmj") pod "04045f92-3e45-49e1-b160-44d9435fe0de" (UID: "04045f92-3e45-49e1-b160-44d9435fe0de"). InnerVolumeSpecName "kube-api-access-2fjmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:52:37.157165 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.157134 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fjmj\" (UniqueName: \"kubernetes.io/projected/04045f92-3e45-49e1-b160-44d9435fe0de-kube-api-access-2fjmj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:52:37.157165 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.157161 2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04045f92-3e45-49e1-b160-44d9435fe0de-config-file\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:52:37.200456 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.200426 2569 generic.go:356] "Generic (PLEG): container finished" podID="04045f92-3e45-49e1-b160-44d9435fe0de" containerID="1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b" exitCode=0 Jul 21 13:52:37.200625 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.200467 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" Jul 21 13:52:37.200625 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.200520 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" event={"ID":"04045f92-3e45-49e1-b160-44d9435fe0de","Type":"ContainerDied","Data":"1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b"} Jul 21 13:52:37.200625 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.200586 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cmvrj" event={"ID":"04045f92-3e45-49e1-b160-44d9435fe0de","Type":"ContainerDied","Data":"b0d3a09ea1669851f1f486fbcf5156ce7c30e1e68a3c3f2a6e89290b6777f2a2"} Jul 21 13:52:37.200625 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.200610 2569 scope.go:117] "RemoveContainer" containerID="1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b" Jul 21 13:52:37.208694 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.208680 2569 scope.go:117] "RemoveContainer" containerID="1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b" Jul 21 13:52:37.208927 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:52:37.208909 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b\": container with ID starting with 1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b not found: ID does not exist" containerID="1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b" Jul 21 13:52:37.208970 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.208934 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b"} err="failed to get container status \"1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b\": rpc error: code = NotFound desc = could not find container \"1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b\": container with ID starting with 1c5ecb7e886def6d39a6870fea0b72e80cace7bdcf15b45ea618c379a365058b not found: ID does not exist" Jul 21 13:52:37.222256 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.222235 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cmvrj"] Jul 21 13:52:37.226351 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:37.226330 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cmvrj"] Jul 21 13:52:38.796090 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:38.796060 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04045f92-3e45-49e1-b160-44d9435fe0de" path="/var/lib/kubelet/pods/04045f92-3e45-49e1-b160-44d9435fe0de/volumes" Jul 21 13:52:42.822067 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.822032 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9"] Jul 21 13:52:42.822482 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.822464 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04045f92-3e45-49e1-b160-44d9435fe0de" containerName="limitador" Jul 21 13:52:42.822482 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.822481 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04045f92-3e45-49e1-b160-44d9435fe0de" containerName="limitador" Jul 21 13:52:42.822640 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.822583 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="04045f92-3e45-49e1-b160-44d9435fe0de" containerName="limitador" Jul 21 13:52:42.826243 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.826227 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:42.829703 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.829681 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 13:52:42.829792 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.829681 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 13:52:42.829979 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.829964 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 13:52:42.830738 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.830721 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-khq2p\"" Jul 21 13:52:42.836455 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.836434 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9"] Jul 21 13:52:42.907375 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.907343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxj5r\" (UniqueName: \"kubernetes.io/projected/c5ee636e-d59b-4353-9f9c-944f80a5cf47-kube-api-access-rxj5r\") pod \"postgres-69fb4594cf-nk8k9\" (UID: \"c5ee636e-d59b-4353-9f9c-944f80a5cf47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:42.907533 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:42.907381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5ee636e-d59b-4353-9f9c-944f80a5cf47-data\") pod \"postgres-69fb4594cf-nk8k9\" (UID: \"c5ee636e-d59b-4353-9f9c-944f80a5cf47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:43.008170 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:43.008135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxj5r\" (UniqueName: \"kubernetes.io/projected/c5ee636e-d59b-4353-9f9c-944f80a5cf47-kube-api-access-rxj5r\") pod \"postgres-69fb4594cf-nk8k9\" (UID: \"c5ee636e-d59b-4353-9f9c-944f80a5cf47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:43.008332 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:43.008175 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5ee636e-d59b-4353-9f9c-944f80a5cf47-data\") pod \"postgres-69fb4594cf-nk8k9\" (UID: \"c5ee636e-d59b-4353-9f9c-944f80a5cf47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:43.008517 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:43.008501 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5ee636e-d59b-4353-9f9c-944f80a5cf47-data\") pod \"postgres-69fb4594cf-nk8k9\" (UID: \"c5ee636e-d59b-4353-9f9c-944f80a5cf47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:43.020271 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:43.020237 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxj5r\" (UniqueName: \"kubernetes.io/projected/c5ee636e-d59b-4353-9f9c-944f80a5cf47-kube-api-access-rxj5r\") pod \"postgres-69fb4594cf-nk8k9\" (UID: \"c5ee636e-d59b-4353-9f9c-944f80a5cf47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:43.137274 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:43.137192 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:43.266784 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:43.266760 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9"] Jul 21 13:52:43.269025 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:52:43.268999 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ee636e_d59b_4353_9f9c_944f80a5cf47.slice/crio-88a21ed1f325b5c26ec71bb6266349c2a98ec11893e3133b29beaef88a3d504c WatchSource:0}: Error finding container 88a21ed1f325b5c26ec71bb6266349c2a98ec11893e3133b29beaef88a3d504c: Status 404 returned error can't find the container with id 88a21ed1f325b5c26ec71bb6266349c2a98ec11893e3133b29beaef88a3d504c Jul 21 13:52:44.245119 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:44.245058 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" event={"ID":"c5ee636e-d59b-4353-9f9c-944f80a5cf47","Type":"ContainerStarted","Data":"88a21ed1f325b5c26ec71bb6266349c2a98ec11893e3133b29beaef88a3d504c"} Jul 21 13:52:48.268174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:48.268141 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" event={"ID":"c5ee636e-d59b-4353-9f9c-944f80a5cf47","Type":"ContainerStarted","Data":"7c62328ac1831e5f23a1582252093efd9d048056f9c4c3c7d4d5d50fdaefd343"} Jul 21 13:52:48.268525 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:48.268276 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:52:48.305201 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:48.305153 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" podStartSLOduration=1.724365271 podStartE2EDuration="6.30514022s" totalImagesPullingTime="4.580774949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:52:48.304789564 +0000 UTC m=+501.342859485" watchObservedRunningTime="2026-07-21 13:52:48.30514022 +0000 UTC m=+501.343210164" Jul 21 13:52:54.301475 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:52:54.301443 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk8k9" Jul 21 13:53:02.232516 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.232485 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-858c6765cb-xvkrw"] Jul 21 13:53:02.239798 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.239778 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:02.243182 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.243157 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 13:53:02.243182 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.243194 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xzg2g\"" Jul 21 13:53:02.243368 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.243226 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 13:53:02.255142 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.253165 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-858c6765cb-xvkrw"] Jul 21 13:53:02.291080 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.291052 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9404f70b-93f4-4442-a72e-c00fc202c62b-webhook-cert\") pod \"maas-controller-858c6765cb-xvkrw\" (UID: \"9404f70b-93f4-4442-a72e-c00fc202c62b\") " pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:02.291225 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.291107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26h5t\" (UniqueName: \"kubernetes.io/projected/9404f70b-93f4-4442-a72e-c00fc202c62b-kube-api-access-26h5t\") pod \"maas-controller-858c6765cb-xvkrw\" (UID: \"9404f70b-93f4-4442-a72e-c00fc202c62b\") " pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:02.391616 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.391574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9404f70b-93f4-4442-a72e-c00fc202c62b-webhook-cert\") pod \"maas-controller-858c6765cb-xvkrw\" (UID: \"9404f70b-93f4-4442-a72e-c00fc202c62b\") " pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:02.391802 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.391743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26h5t\" (UniqueName: \"kubernetes.io/projected/9404f70b-93f4-4442-a72e-c00fc202c62b-kube-api-access-26h5t\") pod \"maas-controller-858c6765cb-xvkrw\" (UID: \"9404f70b-93f4-4442-a72e-c00fc202c62b\") " pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:02.394155 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.394126 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9404f70b-93f4-4442-a72e-c00fc202c62b-webhook-cert\") pod \"maas-controller-858c6765cb-xvkrw\" (UID: \"9404f70b-93f4-4442-a72e-c00fc202c62b\") " pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:02.402089 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.402063 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26h5t\" (UniqueName: \"kubernetes.io/projected/9404f70b-93f4-4442-a72e-c00fc202c62b-kube-api-access-26h5t\") pod \"maas-controller-858c6765cb-xvkrw\" (UID: \"9404f70b-93f4-4442-a72e-c00fc202c62b\") " pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:02.477559 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.477508 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6bd44f67-qlw9d"] Jul 21 13:53:02.481185 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.481168 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:02.502397 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.502374 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-qlw9d"] Jul 21 13:53:02.554409 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.554381 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:02.593557 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.593509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02d2f111-a6bb-4490-b456-46c8d4abf46a-webhook-cert\") pod \"maas-controller-b6bd44f67-qlw9d\" (UID: \"02d2f111-a6bb-4490-b456-46c8d4abf46a\") " pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:02.593701 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.593679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvjvz\" (UniqueName: \"kubernetes.io/projected/02d2f111-a6bb-4490-b456-46c8d4abf46a-kube-api-access-zvjvz\") pod \"maas-controller-b6bd44f67-qlw9d\" (UID: \"02d2f111-a6bb-4490-b456-46c8d4abf46a\") " pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:02.690838 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.690811 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-858c6765cb-xvkrw"] Jul 21 13:53:02.692924 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:53:02.692896 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9404f70b_93f4_4442_a72e_c00fc202c62b.slice/crio-6b889c8d1bfdf0201bb4883fd5a49e3a99685edb855dd07f942910e8e2cd996d WatchSource:0}: Error finding container 6b889c8d1bfdf0201bb4883fd5a49e3a99685edb855dd07f942910e8e2cd996d: Status 404 returned error can't find the container with id 6b889c8d1bfdf0201bb4883fd5a49e3a99685edb855dd07f942910e8e2cd996d Jul 21 13:53:02.694324 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.694301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02d2f111-a6bb-4490-b456-46c8d4abf46a-webhook-cert\") pod \"maas-controller-b6bd44f67-qlw9d\" (UID: \"02d2f111-a6bb-4490-b456-46c8d4abf46a\") " pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:02.694416 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.694390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvjvz\" (UniqueName: \"kubernetes.io/projected/02d2f111-a6bb-4490-b456-46c8d4abf46a-kube-api-access-zvjvz\") pod \"maas-controller-b6bd44f67-qlw9d\" (UID: \"02d2f111-a6bb-4490-b456-46c8d4abf46a\") " pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:02.696874 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.696845 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02d2f111-a6bb-4490-b456-46c8d4abf46a-webhook-cert\") pod \"maas-controller-b6bd44f67-qlw9d\" (UID: \"02d2f111-a6bb-4490-b456-46c8d4abf46a\") " pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:02.705349 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.705325 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvjvz\" (UniqueName: \"kubernetes.io/projected/02d2f111-a6bb-4490-b456-46c8d4abf46a-kube-api-access-zvjvz\") pod \"maas-controller-b6bd44f67-qlw9d\" (UID: \"02d2f111-a6bb-4490-b456-46c8d4abf46a\") " pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:02.791103 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.791026 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:02.932151 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:02.932127 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-qlw9d"] Jul 21 13:53:02.933345 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:53:02.933318 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d2f111_a6bb_4490_b456_46c8d4abf46a.slice/crio-a364ae2001b1ce04ca02f00115dc46d0e207daf955916dbce7d9e71bdf15a8d7 WatchSource:0}: Error finding container a364ae2001b1ce04ca02f00115dc46d0e207daf955916dbce7d9e71bdf15a8d7: Status 404 returned error can't find the container with id a364ae2001b1ce04ca02f00115dc46d0e207daf955916dbce7d9e71bdf15a8d7 Jul 21 13:53:03.340524 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:03.340492 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" event={"ID":"02d2f111-a6bb-4490-b456-46c8d4abf46a","Type":"ContainerStarted","Data":"a364ae2001b1ce04ca02f00115dc46d0e207daf955916dbce7d9e71bdf15a8d7"} Jul 21 13:53:03.341571 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:03.341524 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858c6765cb-xvkrw" event={"ID":"9404f70b-93f4-4442-a72e-c00fc202c62b","Type":"ContainerStarted","Data":"6b889c8d1bfdf0201bb4883fd5a49e3a99685edb855dd07f942910e8e2cd996d"} Jul 21 13:53:05.356127 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:05.356084 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858c6765cb-xvkrw" event={"ID":"9404f70b-93f4-4442-a72e-c00fc202c62b","Type":"ContainerStarted","Data":"c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4"} Jul 21 13:53:05.356562 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:05.356197 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:05.357902 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:05.357868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" event={"ID":"02d2f111-a6bb-4490-b456-46c8d4abf46a","Type":"ContainerStarted","Data":"eb0d46ba420ad0b58a756812c339abf78d02790056f168473ae3da80fce65ba8"} Jul 21 13:53:05.358007 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:05.357946 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:05.378715 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:05.378662 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-858c6765cb-xvkrw" podStartSLOduration=0.794294549 podStartE2EDuration="3.378640616s" totalImagesPullingTime="2.584346067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:53:05.37860089 +0000 UTC m=+518.416670812" watchObservedRunningTime="2026-07-21 13:53:05.378640616 +0000 UTC m=+518.416710536" Jul 21 13:53:05.410066 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:05.410015 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" podStartSLOduration=1.0725809530000001 podStartE2EDuration="3.410000983s" totalImagesPullingTime="2.33742003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:53:05.407872813 +0000 UTC m=+518.445942737" watchObservedRunningTime="2026-07-21 13:53:05.410000983 +0000 UTC m=+518.448070930" Jul 21 13:53:16.366025 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.365988 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:16.366516 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.366078 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:53:16.452759 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.452721 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-858c6765cb-xvkrw"] Jul 21 13:53:16.452966 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.452939 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-858c6765cb-xvkrw" podUID="9404f70b-93f4-4442-a72e-c00fc202c62b" containerName="manager" containerID="cri-o://c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4" gracePeriod=30 Jul 21 13:53:16.700835 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.700810 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:16.822444 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.822417 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26h5t\" (UniqueName: \"kubernetes.io/projected/9404f70b-93f4-4442-a72e-c00fc202c62b-kube-api-access-26h5t\") pod \"9404f70b-93f4-4442-a72e-c00fc202c62b\" (UID: \"9404f70b-93f4-4442-a72e-c00fc202c62b\") " Jul 21 13:53:16.822629 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.822459 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9404f70b-93f4-4442-a72e-c00fc202c62b-webhook-cert\") pod \"9404f70b-93f4-4442-a72e-c00fc202c62b\" (UID: \"9404f70b-93f4-4442-a72e-c00fc202c62b\") " Jul 21 13:53:16.824497 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.824463 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9404f70b-93f4-4442-a72e-c00fc202c62b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9404f70b-93f4-4442-a72e-c00fc202c62b" (UID: "9404f70b-93f4-4442-a72e-c00fc202c62b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:53:16.824656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.824632 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9404f70b-93f4-4442-a72e-c00fc202c62b-kube-api-access-26h5t" (OuterVolumeSpecName: "kube-api-access-26h5t") pod "9404f70b-93f4-4442-a72e-c00fc202c62b" (UID: "9404f70b-93f4-4442-a72e-c00fc202c62b"). InnerVolumeSpecName "kube-api-access-26h5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:53:16.924022 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.923745 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26h5t\" (UniqueName: \"kubernetes.io/projected/9404f70b-93f4-4442-a72e-c00fc202c62b-kube-api-access-26h5t\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:53:16.924022 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:16.923778 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9404f70b-93f4-4442-a72e-c00fc202c62b-webhook-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:53:17.432688 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.432648 2569 generic.go:356] "Generic (PLEG): container finished" podID="9404f70b-93f4-4442-a72e-c00fc202c62b" containerID="c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4" exitCode=0 Jul 21 13:53:17.432688 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.432687 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858c6765cb-xvkrw" Jul 21 13:53:17.433194 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.432732 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858c6765cb-xvkrw" event={"ID":"9404f70b-93f4-4442-a72e-c00fc202c62b","Type":"ContainerDied","Data":"c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4"} Jul 21 13:53:17.433194 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.432794 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858c6765cb-xvkrw" event={"ID":"9404f70b-93f4-4442-a72e-c00fc202c62b","Type":"ContainerDied","Data":"6b889c8d1bfdf0201bb4883fd5a49e3a99685edb855dd07f942910e8e2cd996d"} Jul 21 13:53:17.433194 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.432825 2569 scope.go:117] "RemoveContainer" containerID="c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4" Jul 21 13:53:17.444974 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.444952 2569 scope.go:117] "RemoveContainer" containerID="c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4" Jul 21 13:53:17.445258 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:53:17.445239 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4\": container with ID starting with c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4 not found: ID does not exist" containerID="c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4" Jul 21 13:53:17.445309 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.445267 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4"} err="failed to get container status \"c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4\": rpc error: code = NotFound desc = could not find container \"c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4\": container with ID starting with c3c2bd22885b28d7cfa67a0e5d8b4ba032faa6fa5590ecf41c70b22c44f089d4 not found: ID does not exist" Jul 21 13:53:17.457793 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.457767 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-858c6765cb-xvkrw"] Jul 21 13:53:17.462758 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:17.462736 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-858c6765cb-xvkrw"] Jul 21 13:53:18.795959 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:18.795928 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9404f70b-93f4-4442-a72e-c00fc202c62b" path="/var/lib/kubelet/pods/9404f70b-93f4-4442-a72e-c00fc202c62b/volumes" Jul 21 13:53:40.085001 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.084969 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw"] Jul 21 13:53:40.085466 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.085349 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9404f70b-93f4-4442-a72e-c00fc202c62b" containerName="manager" Jul 21 13:53:40.085466 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.085361 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9404f70b-93f4-4442-a72e-c00fc202c62b" containerName="manager" Jul 21 13:53:40.085466 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.085447 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9404f70b-93f4-4442-a72e-c00fc202c62b" containerName="manager" Jul 21 13:53:40.092426 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.092405 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.096716 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.096688 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nhfmq\"" Jul 21 13:53:40.096716 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.096709 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 13:53:40.108062 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.108036 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw"] Jul 21 13:53:40.139738 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.139704 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18eb1fed-0663-4034-8a5e-59ea83a6469e-maas-api-tls\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.140046 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.139789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxpdh\" (UniqueName: \"kubernetes.io/projected/18eb1fed-0663-4034-8a5e-59ea83a6469e-kube-api-access-jxpdh\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.140046 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.139839 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18eb1fed-0663-4034-8a5e-59ea83a6469e-openshift-service-ca\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.240825 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.240791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxpdh\" (UniqueName: \"kubernetes.io/projected/18eb1fed-0663-4034-8a5e-59ea83a6469e-kube-api-access-jxpdh\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.241008 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.240842 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18eb1fed-0663-4034-8a5e-59ea83a6469e-openshift-service-ca\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.241008 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.240901 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18eb1fed-0663-4034-8a5e-59ea83a6469e-maas-api-tls\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.241482 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.241457 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18eb1fed-0663-4034-8a5e-59ea83a6469e-openshift-service-ca\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.243255 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.243234 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18eb1fed-0663-4034-8a5e-59ea83a6469e-maas-api-tls\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.256858 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.256807 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxpdh\" (UniqueName: \"kubernetes.io/projected/18eb1fed-0663-4034-8a5e-59ea83a6469e-kube-api-access-jxpdh\") pod \"maas-api-745854dcc7-jnkcw\" (UID: \"18eb1fed-0663-4034-8a5e-59ea83a6469e\") " pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.403514 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.403431 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:40.558995 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.558972 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw"] Jul 21 13:53:40.560988 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:53:40.560954 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18eb1fed_0663_4034_8a5e_59ea83a6469e.slice/crio-defa2d0b997bc88437c971e7748af4415641a0f5fb6a6a0c0fd4e08f00a3da23 WatchSource:0}: Error finding container defa2d0b997bc88437c971e7748af4415641a0f5fb6a6a0c0fd4e08f00a3da23: Status 404 returned error can't find the container with id defa2d0b997bc88437c971e7748af4415641a0f5fb6a6a0c0fd4e08f00a3da23 Jul 21 13:53:40.940205 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.940173 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-prk4n"] Jul 21 13:53:40.946460 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.946434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:40.952000 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.951978 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jdr9h\"" Jul 21 13:53:40.952142 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.952123 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 13:53:40.960110 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:40.960088 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-prk4n"] Jul 21 13:53:41.047404 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.047369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssbmz\" (UniqueName: \"kubernetes.io/projected/e9761259-3cef-4858-824d-ba2157670e1f-kube-api-access-ssbmz\") pod \"payload-pre-processing-7677b774bb-prk4n\" (UID: \"e9761259-3cef-4858-824d-ba2157670e1f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:41.047612 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.047426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9761259-3cef-4858-824d-ba2157670e1f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-prk4n\" (UID: \"e9761259-3cef-4858-824d-ba2157670e1f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:41.148684 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.148653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssbmz\" (UniqueName: \"kubernetes.io/projected/e9761259-3cef-4858-824d-ba2157670e1f-kube-api-access-ssbmz\") pod \"payload-pre-processing-7677b774bb-prk4n\" (UID: \"e9761259-3cef-4858-824d-ba2157670e1f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:41.149042 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.148712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9761259-3cef-4858-824d-ba2157670e1f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-prk4n\" (UID: \"e9761259-3cef-4858-824d-ba2157670e1f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:41.149401 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.149381 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9761259-3cef-4858-824d-ba2157670e1f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-prk4n\" (UID: \"e9761259-3cef-4858-824d-ba2157670e1f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:41.160822 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.160803 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssbmz\" (UniqueName: \"kubernetes.io/projected/e9761259-3cef-4858-824d-ba2157670e1f-kube-api-access-ssbmz\") pod \"payload-pre-processing-7677b774bb-prk4n\" (UID: \"e9761259-3cef-4858-824d-ba2157670e1f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:41.258010 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.257967 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:41.413656 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.413622 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-prk4n"] Jul 21 13:53:41.415133 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:53:41.415103 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9761259_3cef_4858_824d_ba2157670e1f.slice/crio-40c9596ccc1233d95017a7676686abb6087858fa2f0e6ecb07c3d8647df8ed6b WatchSource:0}: Error finding container 40c9596ccc1233d95017a7676686abb6087858fa2f0e6ecb07c3d8647df8ed6b: Status 404 returned error can't find the container with id 40c9596ccc1233d95017a7676686abb6087858fa2f0e6ecb07c3d8647df8ed6b Jul 21 13:53:41.551883 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.551782 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" event={"ID":"e9761259-3cef-4858-824d-ba2157670e1f","Type":"ContainerStarted","Data":"40c9596ccc1233d95017a7676686abb6087858fa2f0e6ecb07c3d8647df8ed6b"} Jul 21 13:53:41.553174 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:41.553148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" event={"ID":"18eb1fed-0663-4034-8a5e-59ea83a6469e","Type":"ContainerStarted","Data":"defa2d0b997bc88437c971e7748af4415641a0f5fb6a6a0c0fd4e08f00a3da23"} Jul 21 13:53:42.561950 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:42.561904 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" event={"ID":"18eb1fed-0663-4034-8a5e-59ea83a6469e","Type":"ContainerStarted","Data":"a1275860591f0942e10ab4de16dcf44b2f23fbb019c45b059af706f5cb2fdab1"} Jul 21 13:53:42.562418 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:42.562091 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:42.586629 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:42.586477 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" podStartSLOduration=1.16893366 podStartE2EDuration="2.586458544s" totalImagesPullingTime="1.417524884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:53:42.585472278 +0000 UTC m=+555.623542200" watchObservedRunningTime="2026-07-21 13:53:42.586458544 +0000 UTC m=+555.624528466" Jul 21 13:53:43.570367 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:43.570334 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" event={"ID":"e9761259-3cef-4858-824d-ba2157670e1f","Type":"ContainerStarted","Data":"cc20a86be0231061c0568d86121ba6ce0de3f52e22df2befe2d2fbd81b1687d1"} Jul 21 13:53:43.570867 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:43.570564 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:53:43.591360 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:43.591317 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" podStartSLOduration=1.8305569940000002 podStartE2EDuration="3.591303139s" totalImagesPullingTime="1.760746145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:53:43.588762658 +0000 UTC m=+556.626832576" watchObservedRunningTime="2026-07-21 13:53:43.591303139 +0000 UTC m=+556.629373060" Jul 21 13:53:48.576597 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:48.576561 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-745854dcc7-jnkcw" Jul 21 13:53:54.577368 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:53:54.577336 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-prk4n" Jul 21 13:54:02.923192 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:02.923163 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4"] Jul 21 13:54:02.926889 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:02.926873 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:02.929840 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:02.929820 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 21 13:54:02.931365 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:02.931346 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 13:54:02.931483 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:02.931466 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gr2dk\"" Jul 21 13:54:02.932581 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:02.932566 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 13:54:02.954748 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:02.954728 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4"] Jul 21 13:54:03.061137 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.061103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.061334 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.061156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxmq\" (UniqueName: \"kubernetes.io/projected/895ba857-bb3b-4190-9cd2-f7771c3fda69-kube-api-access-mlxmq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.061334 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.061263 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.061334 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.061306 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.061500 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.061381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.061500 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.061411 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895ba857-bb3b-4190-9cd2-f7771c3fda69-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.162441 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.162412 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.162441 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.162448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.162720 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.162484 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.162720 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.162505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895ba857-bb3b-4190-9cd2-f7771c3fda69-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.162720 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.162569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.162720 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.162625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxmq\" (UniqueName: \"kubernetes.io/projected/895ba857-bb3b-4190-9cd2-f7771c3fda69-kube-api-access-mlxmq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.162940 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.162889 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.162983 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.162964 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.163033 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.163014 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.164904 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.164880 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895ba857-bb3b-4190-9cd2-f7771c3fda69-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.165070 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.165046 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895ba857-bb3b-4190-9cd2-f7771c3fda69-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.173699 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.173632 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxmq\" (UniqueName: \"kubernetes.io/projected/895ba857-bb3b-4190-9cd2-f7771c3fda69-kube-api-access-mlxmq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4\" (UID: \"895ba857-bb3b-4190-9cd2-f7771c3fda69\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.237393 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.237360 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:03.386697 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.386669 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4"] Jul 21 13:54:03.387840 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:54:03.387818 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895ba857_bb3b_4190_9cd2_f7771c3fda69.slice/crio-cb944783d5d01b0b572ea52931e6713b008f4a94b235b390fcd76d46eb537e49 WatchSource:0}: Error finding container cb944783d5d01b0b572ea52931e6713b008f4a94b235b390fcd76d46eb537e49: Status 404 returned error can't find the container with id cb944783d5d01b0b572ea52931e6713b008f4a94b235b390fcd76d46eb537e49 Jul 21 13:54:03.667874 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:03.667837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" event={"ID":"895ba857-bb3b-4190-9cd2-f7771c3fda69","Type":"ContainerStarted","Data":"cb944783d5d01b0b572ea52931e6713b008f4a94b235b390fcd76d46eb537e49"} Jul 21 13:54:09.698975 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:09.698940 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" event={"ID":"895ba857-bb3b-4190-9cd2-f7771c3fda69","Type":"ContainerStarted","Data":"66e862f1acb3183a9c8d9777cd5be3ae7263edfc6b7548c328931bc030213de0"} Jul 21 13:54:09.728488 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:09.728443 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" podStartSLOduration=1.504551503 podStartE2EDuration="7.728427521s" totalImagesPullingTime="6.223876018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:54:09.72671382 +0000 UTC m=+582.764783744" watchObservedRunningTime="2026-07-21 13:54:09.728427521 +0000 UTC m=+582.766497441" Jul 21 13:54:13.237872 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:13.237835 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:13.238354 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:13.237946 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:13.250632 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:13.250597 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:13.728999 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:13.728915 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4" Jul 21 13:54:15.594356 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.594318 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm"] Jul 21 13:54:15.598219 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.598199 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.601470 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.601449 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 13:54:15.617248 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.617223 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm"] Jul 21 13:54:15.775509 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.775479 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38dd0609-7d6a-4195-a32a-86b006a5643c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.775509 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.775512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.775797 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.775531 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.775797 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.775677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.775797 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.775738 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr728\" (UniqueName: \"kubernetes.io/projected/38dd0609-7d6a-4195-a32a-86b006a5643c-kube-api-access-qr728\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.775941 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.775804 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.876976 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.876881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.876976 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.876943 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr728\" (UniqueName: \"kubernetes.io/projected/38dd0609-7d6a-4195-a32a-86b006a5643c-kube-api-access-qr728\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.877209 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.876997 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.877209 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.877070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38dd0609-7d6a-4195-a32a-86b006a5643c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.877209 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.877095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.877209 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.877119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.877403 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.877262 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.877463 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.877441 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.877522 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.877507 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.879350 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.879333 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38dd0609-7d6a-4195-a32a-86b006a5643c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.879459 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.879442 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38dd0609-7d6a-4195-a32a-86b006a5643c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.907308 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.907277 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr728\" (UniqueName: \"kubernetes.io/projected/38dd0609-7d6a-4195-a32a-86b006a5643c-kube-api-access-qr728\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qksgm\" (UID: \"38dd0609-7d6a-4195-a32a-86b006a5643c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:15.909164 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:15.909147 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:16.068475 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:16.068450 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm"] Jul 21 13:54:16.071299 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:54:16.071270 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dd0609_7d6a_4195_a32a_86b006a5643c.slice/crio-58fa50578e42e62a7e5f70dc6ec626786b11adc016b7e257c145ee91e8215ab2 WatchSource:0}: Error finding container 58fa50578e42e62a7e5f70dc6ec626786b11adc016b7e257c145ee91e8215ab2: Status 404 returned error can't find the container with id 58fa50578e42e62a7e5f70dc6ec626786b11adc016b7e257c145ee91e8215ab2 Jul 21 13:54:16.733995 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:16.733970 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" event={"ID":"38dd0609-7d6a-4195-a32a-86b006a5643c","Type":"ContainerStarted","Data":"58fa50578e42e62a7e5f70dc6ec626786b11adc016b7e257c145ee91e8215ab2"} Jul 21 13:54:17.740635 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:17.740591 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" event={"ID":"38dd0609-7d6a-4195-a32a-86b006a5643c","Type":"ContainerStarted","Data":"bef90cbd3245604375886066240d9141b5000d4956f551c58b577d9ed9e9d2c6"} Jul 21 13:54:17.770121 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:17.770066 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" podStartSLOduration=2.111634715 podStartE2EDuration="2.770047792s" totalImagesPullingTime="658.413077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:54:17.768474832 +0000 UTC m=+590.806544754" watchObservedRunningTime="2026-07-21 13:54:17.770047792 +0000 UTC m=+590.808117712" Jul 21 13:54:25.910296 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:25.910260 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:25.910759 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:25.910317 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:25.922726 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:25.922703 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:26.801628 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:26.801602 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qksgm" Jul 21 13:54:30.891036 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.890996 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv"] Jul 21 13:54:30.894483 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.894467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:30.900152 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.900133 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 13:54:30.900263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.900211 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:30.900263 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.900252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:30.900405 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.900382 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzvj\" (UniqueName: \"kubernetes.io/projected/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-kube-api-access-hdzvj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:30.900474 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.900456 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:30.900576 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.900559 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:30.900668 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.900652 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:30.911322 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:30.911298 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv"] Jul 21 13:54:31.002112 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.002301 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002149 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.002301 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.002301 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.002301 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002247 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzvj\" (UniqueName: \"kubernetes.io/projected/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-kube-api-access-hdzvj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.002301 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.002594 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002499 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.002665 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002647 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.002730 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.002712 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.004518 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.004501 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.004831 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.004816 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.020408 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.020388 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzvj\" (UniqueName: \"kubernetes.io/projected/0b70798b-4d10-4051-b63c-0c6d17cbc6b6-kube-api-access-hdzvj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv\" (UID: \"0b70798b-4d10-4051-b63c-0c6d17cbc6b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.204247 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.204161 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:31.342575 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.342549 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv"] Jul 21 13:54:31.344102 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:54:31.344077 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b70798b_4d10_4051_b63c_0c6d17cbc6b6.slice/crio-02d31cbe9e9f045a8f9d5a4305bc658e31d7ac86d9a5463a78bda62a1d36cfbe WatchSource:0}: Error finding container 02d31cbe9e9f045a8f9d5a4305bc658e31d7ac86d9a5463a78bda62a1d36cfbe: Status 404 returned error can't find the container with id 02d31cbe9e9f045a8f9d5a4305bc658e31d7ac86d9a5463a78bda62a1d36cfbe Jul 21 13:54:31.345483 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.345464 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 13:54:31.817783 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.817750 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" event={"ID":"0b70798b-4d10-4051-b63c-0c6d17cbc6b6","Type":"ContainerStarted","Data":"48f3a9681d2d2402b51aebcec8e30261ed724cc8e71fe8861e92a26b4111783a"} Jul 21 13:54:31.817948 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.817798 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" event={"ID":"0b70798b-4d10-4051-b63c-0c6d17cbc6b6","Type":"ContainerStarted","Data":"02d31cbe9e9f045a8f9d5a4305bc658e31d7ac86d9a5463a78bda62a1d36cfbe"} Jul 21 13:54:31.847341 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:31.847243 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" podStartSLOduration=1.623470897 podStartE2EDuration="1.847224681s" totalImagesPullingTime="223.753784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:54:31.845238752 +0000 UTC m=+604.883308700" watchObservedRunningTime="2026-07-21 13:54:31.847224681 +0000 UTC m=+604.885294606" Jul 21 13:54:41.204532 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:41.204494 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:41.204971 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:41.204656 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:41.217204 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:41.217182 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:54:41.876515 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:54:41.876483 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv" Jul 21 13:56:42.350913 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.350878 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-qlw9d"] Jul 21 13:56:42.351431 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.351143 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" podUID="02d2f111-a6bb-4490-b456-46c8d4abf46a" containerName="manager" containerID="cri-o://eb0d46ba420ad0b58a756812c339abf78d02790056f168473ae3da80fce65ba8" gracePeriod=30 Jul 21 13:56:42.467945 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.467915 2569 generic.go:356] "Generic (PLEG): container finished" podID="02d2f111-a6bb-4490-b456-46c8d4abf46a" containerID="eb0d46ba420ad0b58a756812c339abf78d02790056f168473ae3da80fce65ba8" exitCode=0 Jul 21 13:56:42.468091 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.467984 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" event={"ID":"02d2f111-a6bb-4490-b456-46c8d4abf46a","Type":"ContainerDied","Data":"eb0d46ba420ad0b58a756812c339abf78d02790056f168473ae3da80fce65ba8"} Jul 21 13:56:42.597218 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.597191 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:56:42.645862 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.645785 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02d2f111-a6bb-4490-b456-46c8d4abf46a-webhook-cert\") pod \"02d2f111-a6bb-4490-b456-46c8d4abf46a\" (UID: \"02d2f111-a6bb-4490-b456-46c8d4abf46a\") " Jul 21 13:56:42.646018 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.645867 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvjvz\" (UniqueName: \"kubernetes.io/projected/02d2f111-a6bb-4490-b456-46c8d4abf46a-kube-api-access-zvjvz\") pod \"02d2f111-a6bb-4490-b456-46c8d4abf46a\" (UID: \"02d2f111-a6bb-4490-b456-46c8d4abf46a\") " Jul 21 13:56:42.647852 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.647815 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d2f111-a6bb-4490-b456-46c8d4abf46a-kube-api-access-zvjvz" (OuterVolumeSpecName: "kube-api-access-zvjvz") pod "02d2f111-a6bb-4490-b456-46c8d4abf46a" (UID: "02d2f111-a6bb-4490-b456-46c8d4abf46a"). InnerVolumeSpecName "kube-api-access-zvjvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:56:42.647962 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.647864 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d2f111-a6bb-4490-b456-46c8d4abf46a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "02d2f111-a6bb-4490-b456-46c8d4abf46a" (UID: "02d2f111-a6bb-4490-b456-46c8d4abf46a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:56:42.746797 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.746759 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02d2f111-a6bb-4490-b456-46c8d4abf46a-webhook-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:56:42.746797 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:42.746791 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvjvz\" (UniqueName: \"kubernetes.io/projected/02d2f111-a6bb-4490-b456-46c8d4abf46a-kube-api-access-zvjvz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:56:43.474297 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:43.474263 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" Jul 21 13:56:43.474735 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:43.474258 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-qlw9d" event={"ID":"02d2f111-a6bb-4490-b456-46c8d4abf46a","Type":"ContainerDied","Data":"a364ae2001b1ce04ca02f00115dc46d0e207daf955916dbce7d9e71bdf15a8d7"} Jul 21 13:56:43.474735 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:43.474417 2569 scope.go:117] "RemoveContainer" containerID="eb0d46ba420ad0b58a756812c339abf78d02790056f168473ae3da80fce65ba8" Jul 21 13:56:43.491684 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:43.491661 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-qlw9d"] Jul 21 13:56:43.498509 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:43.498487 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-qlw9d"] Jul 21 13:56:44.795725 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:56:44.795691 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d2f111-a6bb-4490-b456-46c8d4abf46a" path="/var/lib/kubelet/pods/02d2f111-a6bb-4490-b456-46c8d4abf46a/volumes" Jul 21 13:57:44.326852 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.326820 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6bd44f67-8kz9j"] Jul 21 13:57:44.327234 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.327222 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d2f111-a6bb-4490-b456-46c8d4abf46a" containerName="manager" Jul 21 13:57:44.327234 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.327233 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d2f111-a6bb-4490-b456-46c8d4abf46a" containerName="manager" Jul 21 13:57:44.327310 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.327304 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d2f111-a6bb-4490-b456-46c8d4abf46a" containerName="manager" Jul 21 13:57:44.331820 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.331804 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:44.334673 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.334654 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 13:57:44.334775 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.334652 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xzg2g\"" Jul 21 13:57:44.335684 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.335671 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 13:57:44.342650 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.342624 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-8kz9j"] Jul 21 13:57:44.440925 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.440887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba18a4f6-148a-4bbf-a11a-49a17373387a-webhook-cert\") pod \"maas-controller-b6bd44f67-8kz9j\" (UID: \"ba18a4f6-148a-4bbf-a11a-49a17373387a\") " pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:44.441105 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.440941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rbr\" (UniqueName: \"kubernetes.io/projected/ba18a4f6-148a-4bbf-a11a-49a17373387a-kube-api-access-k6rbr\") pod \"maas-controller-b6bd44f67-8kz9j\" (UID: \"ba18a4f6-148a-4bbf-a11a-49a17373387a\") " pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:44.542378 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.542336 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba18a4f6-148a-4bbf-a11a-49a17373387a-webhook-cert\") pod \"maas-controller-b6bd44f67-8kz9j\" (UID: \"ba18a4f6-148a-4bbf-a11a-49a17373387a\") " pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:44.542571 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.542493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rbr\" (UniqueName: \"kubernetes.io/projected/ba18a4f6-148a-4bbf-a11a-49a17373387a-kube-api-access-k6rbr\") pod \"maas-controller-b6bd44f67-8kz9j\" (UID: \"ba18a4f6-148a-4bbf-a11a-49a17373387a\") " pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:44.544788 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.544761 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba18a4f6-148a-4bbf-a11a-49a17373387a-webhook-cert\") pod \"maas-controller-b6bd44f67-8kz9j\" (UID: \"ba18a4f6-148a-4bbf-a11a-49a17373387a\") " pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:44.551178 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.551161 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rbr\" (UniqueName: \"kubernetes.io/projected/ba18a4f6-148a-4bbf-a11a-49a17373387a-kube-api-access-k6rbr\") pod \"maas-controller-b6bd44f67-8kz9j\" (UID: \"ba18a4f6-148a-4bbf-a11a-49a17373387a\") " pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:44.644095 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.644011 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:44.775074 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:44.775046 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-8kz9j"] Jul 21 13:57:44.776315 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:57:44.776283 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba18a4f6_148a_4bbf_a11a_49a17373387a.slice/crio-341491a5c09336c15a1793114f5d60fd4504030f1f31c555f9faca54d16db486 WatchSource:0}: Error finding container 341491a5c09336c15a1793114f5d60fd4504030f1f31c555f9faca54d16db486: Status 404 returned error can't find the container with id 341491a5c09336c15a1793114f5d60fd4504030f1f31c555f9faca54d16db486 Jul 21 13:57:45.783500 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:45.783467 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" event={"ID":"ba18a4f6-148a-4bbf-a11a-49a17373387a","Type":"ContainerStarted","Data":"341491a5c09336c15a1793114f5d60fd4504030f1f31c555f9faca54d16db486"} Jul 21 13:57:47.802123 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:47.802086 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" event={"ID":"ba18a4f6-148a-4bbf-a11a-49a17373387a","Type":"ContainerStarted","Data":"c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2"} Jul 21 13:57:47.802501 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:47.802148 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:57:47.822802 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:47.822755 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" podStartSLOduration=1.5609016009999999 podStartE2EDuration="3.822740587s" totalImagesPullingTime="2.261838986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:57:47.821678088 +0000 UTC m=+800.859748012" watchObservedRunningTime="2026-07-21 13:57:47.822740587 +0000 UTC m=+800.860810507" Jul 21 13:57:58.809079 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:57:58.809009 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:58:03.502070 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.502033 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-8kz9j"] Jul 21 13:58:03.502468 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.502239 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" podUID="ba18a4f6-148a-4bbf-a11a-49a17373387a" containerName="manager" containerID="cri-o://c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2" gracePeriod=30 Jul 21 13:58:03.755201 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.755142 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:58:03.820557 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.820512 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba18a4f6-148a-4bbf-a11a-49a17373387a-webhook-cert\") pod \"ba18a4f6-148a-4bbf-a11a-49a17373387a\" (UID: \"ba18a4f6-148a-4bbf-a11a-49a17373387a\") " Jul 21 13:58:03.820737 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.820660 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6rbr\" (UniqueName: \"kubernetes.io/projected/ba18a4f6-148a-4bbf-a11a-49a17373387a-kube-api-access-k6rbr\") pod \"ba18a4f6-148a-4bbf-a11a-49a17373387a\" (UID: \"ba18a4f6-148a-4bbf-a11a-49a17373387a\") " Jul 21 13:58:03.822528 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.822481 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba18a4f6-148a-4bbf-a11a-49a17373387a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ba18a4f6-148a-4bbf-a11a-49a17373387a" (UID: "ba18a4f6-148a-4bbf-a11a-49a17373387a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:58:03.822669 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.822562 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba18a4f6-148a-4bbf-a11a-49a17373387a-kube-api-access-k6rbr" (OuterVolumeSpecName: "kube-api-access-k6rbr") pod "ba18a4f6-148a-4bbf-a11a-49a17373387a" (UID: "ba18a4f6-148a-4bbf-a11a-49a17373387a"). InnerVolumeSpecName "kube-api-access-k6rbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:58:03.880169 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.880139 2569 generic.go:356] "Generic (PLEG): container finished" podID="ba18a4f6-148a-4bbf-a11a-49a17373387a" containerID="c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2" exitCode=0 Jul 21 13:58:03.880299 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.880188 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" Jul 21 13:58:03.880299 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.880224 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" event={"ID":"ba18a4f6-148a-4bbf-a11a-49a17373387a","Type":"ContainerDied","Data":"c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2"} Jul 21 13:58:03.880299 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.880279 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-8kz9j" event={"ID":"ba18a4f6-148a-4bbf-a11a-49a17373387a","Type":"ContainerDied","Data":"341491a5c09336c15a1793114f5d60fd4504030f1f31c555f9faca54d16db486"} Jul 21 13:58:03.880403 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.880308 2569 scope.go:117] "RemoveContainer" containerID="c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2" Jul 21 13:58:03.889003 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.888983 2569 scope.go:117] "RemoveContainer" containerID="c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2" Jul 21 13:58:03.889238 ip-10-0-129-45 kubenswrapper[2569]: E0721 13:58:03.889219 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2\": container with ID starting with c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2 not found: ID does not exist" containerID="c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2" Jul 21 13:58:03.889287 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.889247 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2"} err="failed to get container status \"c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2\": rpc error: code = NotFound desc = could not find container \"c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2\": container with ID starting with c74a3fdcca70582a687b70843179e3fee727df15e728a84820867c06c07102f2 not found: ID does not exist" Jul 21 13:58:03.909298 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.909274 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-8kz9j"] Jul 21 13:58:03.921662 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.921641 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba18a4f6-148a-4bbf-a11a-49a17373387a-webhook-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:58:03.921755 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.921667 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rbr\" (UniqueName: \"kubernetes.io/projected/ba18a4f6-148a-4bbf-a11a-49a17373387a-kube-api-access-k6rbr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 13:58:03.921798 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:03.921762 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-8kz9j"] Jul 21 13:58:04.751813 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.751779 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6bd44f67-rh5qr"] Jul 21 13:58:04.752199 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.752167 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba18a4f6-148a-4bbf-a11a-49a17373387a" containerName="manager" Jul 21 13:58:04.752286 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.752200 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba18a4f6-148a-4bbf-a11a-49a17373387a" containerName="manager" Jul 21 13:58:04.752286 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.752284 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba18a4f6-148a-4bbf-a11a-49a17373387a" containerName="manager" Jul 21 13:58:04.756753 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.756735 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:04.760247 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.760222 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 13:58:04.761247 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.761231 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xzg2g\"" Jul 21 13:58:04.761336 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.761297 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 13:58:04.777016 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.776990 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-rh5qr"] Jul 21 13:58:04.796805 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.796778 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba18a4f6-148a-4bbf-a11a-49a17373387a" path="/var/lib/kubelet/pods/ba18a4f6-148a-4bbf-a11a-49a17373387a/volumes" Jul 21 13:58:04.830611 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.830573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64f132c4-bccf-47f2-9c9e-4205e33f5a83-webhook-cert\") pod \"maas-controller-b6bd44f67-rh5qr\" (UID: \"64f132c4-bccf-47f2-9c9e-4205e33f5a83\") " pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:04.830742 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.830634 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dklf\" (UniqueName: \"kubernetes.io/projected/64f132c4-bccf-47f2-9c9e-4205e33f5a83-kube-api-access-9dklf\") pod \"maas-controller-b6bd44f67-rh5qr\" (UID: \"64f132c4-bccf-47f2-9c9e-4205e33f5a83\") " pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:04.931426 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.931386 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64f132c4-bccf-47f2-9c9e-4205e33f5a83-webhook-cert\") pod \"maas-controller-b6bd44f67-rh5qr\" (UID: \"64f132c4-bccf-47f2-9c9e-4205e33f5a83\") " pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:04.931426 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.931423 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dklf\" (UniqueName: \"kubernetes.io/projected/64f132c4-bccf-47f2-9c9e-4205e33f5a83-kube-api-access-9dklf\") pod \"maas-controller-b6bd44f67-rh5qr\" (UID: \"64f132c4-bccf-47f2-9c9e-4205e33f5a83\") " pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:04.933702 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.933677 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64f132c4-bccf-47f2-9c9e-4205e33f5a83-webhook-cert\") pod \"maas-controller-b6bd44f67-rh5qr\" (UID: \"64f132c4-bccf-47f2-9c9e-4205e33f5a83\") " pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:04.940642 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:04.940618 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dklf\" (UniqueName: \"kubernetes.io/projected/64f132c4-bccf-47f2-9c9e-4205e33f5a83-kube-api-access-9dklf\") pod \"maas-controller-b6bd44f67-rh5qr\" (UID: \"64f132c4-bccf-47f2-9c9e-4205e33f5a83\") " pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:05.066713 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:05.066634 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:05.204466 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:05.204421 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6bd44f67-rh5qr"] Jul 21 13:58:05.207635 ip-10-0-129-45 kubenswrapper[2569]: W0721 13:58:05.207605 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f132c4_bccf_47f2_9c9e_4205e33f5a83.slice/crio-f5785c61fffb67254aee7effc59d44ae2a623b6ae2a7b28bb23cfc1cbfa9be2e WatchSource:0}: Error finding container f5785c61fffb67254aee7effc59d44ae2a623b6ae2a7b28bb23cfc1cbfa9be2e: Status 404 returned error can't find the container with id f5785c61fffb67254aee7effc59d44ae2a623b6ae2a7b28bb23cfc1cbfa9be2e Jul 21 13:58:05.892224 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:05.892128 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-rh5qr" event={"ID":"64f132c4-bccf-47f2-9c9e-4205e33f5a83","Type":"ContainerStarted","Data":"7e40fdc022ff2ca6a3ff854ed539972f3799c0da2d2ca89311b2d3e7513c386f"} Jul 21 13:58:05.892224 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:05.892180 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6bd44f67-rh5qr" event={"ID":"64f132c4-bccf-47f2-9c9e-4205e33f5a83","Type":"ContainerStarted","Data":"f5785c61fffb67254aee7effc59d44ae2a623b6ae2a7b28bb23cfc1cbfa9be2e"} Jul 21 13:58:05.892678 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:05.892241 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 13:58:05.925944 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:05.925894 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6bd44f67-rh5qr" podStartSLOduration=1.5697563319999999 podStartE2EDuration="1.925878938s" totalImagesPullingTime="356.122606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:58:05.923436746 +0000 UTC m=+818.961506680" watchObservedRunningTime="2026-07-21 13:58:05.925878938 +0000 UTC m=+818.963948856" Jul 21 13:58:16.899573 ip-10-0-129-45 kubenswrapper[2569]: I0721 13:58:16.899505 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6bd44f67-rh5qr" Jul 21 14:20:12.753816 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.753781 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2"] Jul 21 14:20:12.757665 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.757642 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:12.762592 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.762571 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-faeda6\"/\"test-model-faeda6-kserve-self-signed-certs\"" Jul 21 14:20:12.762592 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.762589 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-faeda6\"/\"openshift-service-ca.crt\"" Jul 21 14:20:12.763700 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.763680 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-faeda6\"/\"default-dockercfg-2n5hf\"" Jul 21 14:20:12.763813 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.763681 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-faeda6\"/\"kube-root-ca.crt\"" Jul 21 14:20:12.770355 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.770333 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2"] Jul 21 14:20:12.905770 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.905709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-tmp-dir\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:12.905770 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.905778 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57s66\" (UniqueName: \"kubernetes.io/projected/25864883-7be7-4193-b180-217effc2773b-kube-api-access-57s66\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:12.906047 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.905855 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-home\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:12.906047 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.905895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-dshm\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:12.906047 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.905912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25864883-7be7-4193-b180-217effc2773b-tls-certs\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:12.906191 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:12.906048 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-model-cache\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007308 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-model-cache\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007308 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-tmp-dir\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007571 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57s66\" (UniqueName: \"kubernetes.io/projected/25864883-7be7-4193-b180-217effc2773b-kube-api-access-57s66\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007571 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007483 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-home\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007571 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-dshm\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007747 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25864883-7be7-4193-b180-217effc2773b-tls-certs\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007747 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007650 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-model-cache\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007747 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007714 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-tmp-dir\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.007853 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.007837 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-home\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.009919 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.009885 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-dshm\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.010061 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.010034 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25864883-7be7-4193-b180-217effc2773b-tls-certs\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.016712 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.016690 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57s66\" (UniqueName: \"kubernetes.io/projected/25864883-7be7-4193-b180-217effc2773b-kube-api-access-57s66\") pod \"test-model-faeda6-kserve-66b5cbcdd7-8tjk2\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.067412 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.067378 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:13.199359 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.199331 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2"] Jul 21 14:20:13.201969 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:20:13.201940 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25864883_7be7_4193_b180_217effc2773b.slice/crio-5a48cd9c648d6134de840767f3b2722656fd4255dbd160d88b453f8f0b785d7c WatchSource:0}: Error finding container 5a48cd9c648d6134de840767f3b2722656fd4255dbd160d88b453f8f0b785d7c: Status 404 returned error can't find the container with id 5a48cd9c648d6134de840767f3b2722656fd4255dbd160d88b453f8f0b785d7c Jul 21 14:20:13.368319 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.368278 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" event={"ID":"25864883-7be7-4193-b180-217effc2773b","Type":"ContainerStarted","Data":"89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845"} Jul 21 14:20:13.368319 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.368322 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" event={"ID":"25864883-7be7-4193-b180-217effc2773b","Type":"ContainerStarted","Data":"5a48cd9c648d6134de840767f3b2722656fd4255dbd160d88b453f8f0b785d7c"} Jul 21 14:20:13.385882 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:13.385835 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" podStartSLOduration=1.385816611 podStartE2EDuration="1.385816611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:20:13.385637292 +0000 UTC m=+2146.423707213" watchObservedRunningTime="2026-07-21 14:20:13.385816611 +0000 UTC m=+2146.423886535" Jul 21 14:20:23.067927 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:23.067848 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:23.067927 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:23.067897 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:23.080135 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:23.080114 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:23.428554 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:23.428460 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:20:55.639686 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:20:55.639649 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2"] Jul 21 14:21:10.660049 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.659956 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" podUID="25864883-7be7-4193-b180-217effc2773b" containerName="main" containerID="cri-o://89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845" gracePeriod=45 Jul 21 14:21:10.906656 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.906632 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:21:10.968965 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.968900 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-dshm\") pod \"25864883-7be7-4193-b180-217effc2773b\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " Jul 21 14:21:10.969080 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969012 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25864883-7be7-4193-b180-217effc2773b-tls-certs\") pod \"25864883-7be7-4193-b180-217effc2773b\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " Jul 21 14:21:10.969080 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969046 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-tmp-dir\") pod \"25864883-7be7-4193-b180-217effc2773b\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " Jul 21 14:21:10.969080 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969070 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-home\") pod \"25864883-7be7-4193-b180-217effc2773b\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " Jul 21 14:21:10.969245 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969097 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-model-cache\") pod \"25864883-7be7-4193-b180-217effc2773b\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " Jul 21 14:21:10.969245 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969121 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57s66\" (UniqueName: \"kubernetes.io/projected/25864883-7be7-4193-b180-217effc2773b-kube-api-access-57s66\") pod \"25864883-7be7-4193-b180-217effc2773b\" (UID: \"25864883-7be7-4193-b180-217effc2773b\") " Jul 21 14:21:10.969373 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969337 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-home" (OuterVolumeSpecName: "home") pod "25864883-7be7-4193-b180-217effc2773b" (UID: "25864883-7be7-4193-b180-217effc2773b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:21:10.969373 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969349 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-model-cache" (OuterVolumeSpecName: "model-cache") pod "25864883-7be7-4193-b180-217effc2773b" (UID: "25864883-7be7-4193-b180-217effc2773b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:21:10.969475 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969365 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25864883-7be7-4193-b180-217effc2773b" (UID: "25864883-7be7-4193-b180-217effc2773b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:21:10.969475 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969451 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:21:10.969475 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969469 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:21:10.969652 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.969480 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:21:10.971171 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.971145 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-dshm" (OuterVolumeSpecName: "dshm") pod "25864883-7be7-4193-b180-217effc2773b" (UID: "25864883-7be7-4193-b180-217effc2773b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:21:10.971275 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.971162 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25864883-7be7-4193-b180-217effc2773b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25864883-7be7-4193-b180-217effc2773b" (UID: "25864883-7be7-4193-b180-217effc2773b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:21:10.971275 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:10.971216 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25864883-7be7-4193-b180-217effc2773b-kube-api-access-57s66" (OuterVolumeSpecName: "kube-api-access-57s66") pod "25864883-7be7-4193-b180-217effc2773b" (UID: "25864883-7be7-4193-b180-217effc2773b"). InnerVolumeSpecName "kube-api-access-57s66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:21:11.070962 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.070931 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57s66\" (UniqueName: \"kubernetes.io/projected/25864883-7be7-4193-b180-217effc2773b-kube-api-access-57s66\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:21:11.070962 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.070960 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25864883-7be7-4193-b180-217effc2773b-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:21:11.071137 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.070972 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25864883-7be7-4193-b180-217effc2773b-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:21:11.667915 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.667884 2569 generic.go:356] "Generic (PLEG): container finished" podID="25864883-7be7-4193-b180-217effc2773b" containerID="89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845" exitCode=0 Jul 21 14:21:11.668295 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.667943 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" Jul 21 14:21:11.668295 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.667974 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" event={"ID":"25864883-7be7-4193-b180-217effc2773b","Type":"ContainerDied","Data":"89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845"} Jul 21 14:21:11.668295 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.668020 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2" event={"ID":"25864883-7be7-4193-b180-217effc2773b","Type":"ContainerDied","Data":"5a48cd9c648d6134de840767f3b2722656fd4255dbd160d88b453f8f0b785d7c"} Jul 21 14:21:11.668295 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.668037 2569 scope.go:117] "RemoveContainer" containerID="89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845" Jul 21 14:21:11.677219 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.677202 2569 scope.go:117] "RemoveContainer" containerID="89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845" Jul 21 14:21:11.677464 ip-10-0-129-45 kubenswrapper[2569]: E0721 14:21:11.677438 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845\": container with ID starting with 89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845 not found: ID does not exist" containerID="89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845" Jul 21 14:21:11.677518 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.677464 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845"} err="failed to get container status \"89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845\": rpc error: code = NotFound desc = could not find container \"89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845\": container with ID starting with 89c349701ccba31aefa50bd6eab5fcbce4e5eec326ae26523b3dec9037a55845 not found: ID does not exist" Jul 21 14:21:11.689015 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.688994 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2"] Jul 21 14:21:11.695151 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:11.695132 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-faeda6/test-model-faeda6-kserve-66b5cbcdd7-8tjk2"] Jul 21 14:21:12.796855 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:21:12.796825 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25864883-7be7-4193-b180-217effc2773b" path="/var/lib/kubelet/pods/25864883-7be7-4193-b180-217effc2773b/volumes" Jul 21 14:26:22.181272 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.181236 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j"] Jul 21 14:26:22.181771 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.181656 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25864883-7be7-4193-b180-217effc2773b" containerName="main" Jul 21 14:26:22.181771 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.181669 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="25864883-7be7-4193-b180-217effc2773b" containerName="main" Jul 21 14:26:22.181771 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.181753 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="25864883-7be7-4193-b180-217effc2773b" containerName="main" Jul 21 14:26:22.185054 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.185034 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.187770 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.187747 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67be07\"/\"default-dockercfg-s45j2\"" Jul 21 14:26:22.187950 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.187938 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-67be07\"/\"kube-root-ca.crt\"" Jul 21 14:26:22.188945 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.188910 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-67be07\"/\"openshift-service-ca.crt\"" Jul 21 14:26:22.189068 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.189020 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67be07\"/\"sub-test-model-67be07-kserve-self-signed-certs\"" Jul 21 14:26:22.200072 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.200045 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j"] Jul 21 14:26:22.328831 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.328777 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-dshm\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.328831 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.328839 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-home\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.329047 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.328869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-tmp-dir\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.329047 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.328913 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b99df94-3226-4a3f-9471-125bb7086943-tls-certs\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.329047 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.328944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-model-cache\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.329047 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.328982 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rkx\" (UniqueName: \"kubernetes.io/projected/7b99df94-3226-4a3f-9471-125bb7086943-kube-api-access-d9rkx\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.429789 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.429743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-dshm\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.429789 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.429796 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-home\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.430011 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.429918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-tmp-dir\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.430011 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.429985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b99df94-3226-4a3f-9471-125bb7086943-tls-certs\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.430088 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.430044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-model-cache\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.430125 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.430103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rkx\" (UniqueName: \"kubernetes.io/projected/7b99df94-3226-4a3f-9471-125bb7086943-kube-api-access-d9rkx\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.430176 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.430158 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-home\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.430278 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.430253 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-tmp-dir\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.430436 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.430416 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-model-cache\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.432197 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.432145 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-dshm\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.432304 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.432285 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b99df94-3226-4a3f-9471-125bb7086943-tls-certs\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.439373 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.439347 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rkx\" (UniqueName: \"kubernetes.io/projected/7b99df94-3226-4a3f-9471-125bb7086943-kube-api-access-d9rkx\") pod \"sub-test-model-67be07-kserve-55f445d885-qn92j\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.494820 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.494776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:22.631376 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:22.631344 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j"] Jul 21 14:26:22.633606 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:26:22.633570 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b99df94_3226_4a3f_9471_125bb7086943.slice/crio-a3959c9041ee2d06a0b92f1a37e265e879793b358d90c8ab4a67cad2ab81a6f1 WatchSource:0}: Error finding container a3959c9041ee2d06a0b92f1a37e265e879793b358d90c8ab4a67cad2ab81a6f1: Status 404 returned error can't find the container with id a3959c9041ee2d06a0b92f1a37e265e879793b358d90c8ab4a67cad2ab81a6f1 Jul 21 14:26:23.214566 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:23.214519 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" event={"ID":"7b99df94-3226-4a3f-9471-125bb7086943","Type":"ContainerStarted","Data":"a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb"} Jul 21 14:26:23.214566 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:23.214575 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" event={"ID":"7b99df94-3226-4a3f-9471-125bb7086943","Type":"ContainerStarted","Data":"a3959c9041ee2d06a0b92f1a37e265e879793b358d90c8ab4a67cad2ab81a6f1"} Jul 21 14:26:23.235184 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:23.235133 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" podStartSLOduration=1.235119012 podStartE2EDuration="1.235119012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:26:23.232516606 +0000 UTC m=+2516.270586526" watchObservedRunningTime="2026-07-21 14:26:23.235119012 +0000 UTC m=+2516.273188933" Jul 21 14:26:32.495381 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:32.495349 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:32.495858 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:32.495477 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:32.508013 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:32.507987 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:33.274062 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:33.274032 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:26:47.564077 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.564044 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq"] Jul 21 14:26:47.567837 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.567812 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.570442 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.570421 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52967c\"/\"kube-root-ca.crt\"" Jul 21 14:26:47.571534 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.571514 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52967c\"/\"default-dockercfg-qfcs7\"" Jul 21 14:26:47.571649 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.571514 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52967c\"/\"sub-test-model-52967c-kserve-self-signed-certs\"" Jul 21 14:26:47.571649 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.571583 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52967c\"/\"openshift-service-ca.crt\"" Jul 21 14:26:47.579416 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.579391 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq"] Jul 21 14:26:47.677070 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.677030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-dshm\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.677246 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.677087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-model-cache\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.677246 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.677145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-home\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.677246 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.677202 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1091bd22-b0c9-4407-a925-80dda8a34544-tls-certs\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.677246 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.677221 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8r5\" (UniqueName: \"kubernetes.io/projected/1091bd22-b0c9-4407-a925-80dda8a34544-kube-api-access-vf8r5\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.677386 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.677307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-tmp-dir\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.778609 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.778569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8r5\" (UniqueName: \"kubernetes.io/projected/1091bd22-b0c9-4407-a925-80dda8a34544-kube-api-access-vf8r5\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.778766 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.778621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-tmp-dir\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.778766 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.778716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-dshm\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.778874 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.778765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-model-cache\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.778874 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.778793 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-home\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.778874 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.778846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1091bd22-b0c9-4407-a925-80dda8a34544-tls-certs\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.779134 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.779109 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-tmp-dir\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.779210 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.779159 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-model-cache\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.779210 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.779175 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-home\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.780987 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.780967 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-dshm\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.781245 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.781226 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1091bd22-b0c9-4407-a925-80dda8a34544-tls-certs\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.787285 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.787263 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8r5\" (UniqueName: \"kubernetes.io/projected/1091bd22-b0c9-4407-a925-80dda8a34544-kube-api-access-vf8r5\") pod \"sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:47.880030 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:47.879951 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:48.015405 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:48.015380 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq"] Jul 21 14:26:48.017408 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:26:48.017375 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1091bd22_b0c9_4407_a925_80dda8a34544.slice/crio-c92ad79cf565f8eaea37bd5f4c32212564686cea7a45cc68abc0a362f9e7d3f7 WatchSource:0}: Error finding container c92ad79cf565f8eaea37bd5f4c32212564686cea7a45cc68abc0a362f9e7d3f7: Status 404 returned error can't find the container with id c92ad79cf565f8eaea37bd5f4c32212564686cea7a45cc68abc0a362f9e7d3f7 Jul 21 14:26:48.335330 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:48.335296 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" event={"ID":"1091bd22-b0c9-4407-a925-80dda8a34544","Type":"ContainerStarted","Data":"b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f"} Jul 21 14:26:48.335509 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:48.335338 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" event={"ID":"1091bd22-b0c9-4407-a925-80dda8a34544","Type":"ContainerStarted","Data":"c92ad79cf565f8eaea37bd5f4c32212564686cea7a45cc68abc0a362f9e7d3f7"} Jul 21 14:26:48.354511 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:48.354456 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" podStartSLOduration=1.354441092 podStartE2EDuration="1.354441092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:26:48.352530003 +0000 UTC m=+2541.390599924" watchObservedRunningTime="2026-07-21 14:26:48.354441092 +0000 UTC m=+2541.392511012" Jul 21 14:26:57.880853 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:57.880813 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:57.881339 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:57.880869 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:57.893249 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:57.893223 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:26:58.395162 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:26:58.395135 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:27:35.815704 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:35.815673 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j"] Jul 21 14:27:36.684897 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:36.684863 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq"] Jul 21 14:27:50.836091 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:50.836004 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" podUID="7b99df94-3226-4a3f-9471-125bb7086943" containerName="main" containerID="cri-o://a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb" gracePeriod=45 Jul 21 14:27:51.085160 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.085129 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:27:51.204013 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.203938 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-home\") pod \"7b99df94-3226-4a3f-9471-125bb7086943\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " Jul 21 14:27:51.204173 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204016 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-dshm\") pod \"7b99df94-3226-4a3f-9471-125bb7086943\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " Jul 21 14:27:51.204173 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204046 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-tmp-dir\") pod \"7b99df94-3226-4a3f-9471-125bb7086943\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " Jul 21 14:27:51.204173 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204077 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-model-cache\") pod \"7b99df94-3226-4a3f-9471-125bb7086943\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " Jul 21 14:27:51.204358 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204175 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b99df94-3226-4a3f-9471-125bb7086943-tls-certs\") pod \"7b99df94-3226-4a3f-9471-125bb7086943\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " Jul 21 14:27:51.204358 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204237 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9rkx\" (UniqueName: \"kubernetes.io/projected/7b99df94-3226-4a3f-9471-125bb7086943-kube-api-access-d9rkx\") pod \"7b99df94-3226-4a3f-9471-125bb7086943\" (UID: \"7b99df94-3226-4a3f-9471-125bb7086943\") " Jul 21 14:27:51.204358 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204245 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-home" (OuterVolumeSpecName: "home") pod "7b99df94-3226-4a3f-9471-125bb7086943" (UID: "7b99df94-3226-4a3f-9471-125bb7086943"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:27:51.204517 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204350 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b99df94-3226-4a3f-9471-125bb7086943" (UID: "7b99df94-3226-4a3f-9471-125bb7086943"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:27:51.204625 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204526 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b99df94-3226-4a3f-9471-125bb7086943" (UID: "7b99df94-3226-4a3f-9471-125bb7086943"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:27:51.204714 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204690 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:51.204852 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204717 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:51.204852 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.204732 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:51.206272 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.206251 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-dshm" (OuterVolumeSpecName: "dshm") pod "7b99df94-3226-4a3f-9471-125bb7086943" (UID: "7b99df94-3226-4a3f-9471-125bb7086943"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:27:51.206612 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.206587 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b99df94-3226-4a3f-9471-125bb7086943-kube-api-access-d9rkx" (OuterVolumeSpecName: "kube-api-access-d9rkx") pod "7b99df94-3226-4a3f-9471-125bb7086943" (UID: "7b99df94-3226-4a3f-9471-125bb7086943"). InnerVolumeSpecName "kube-api-access-d9rkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:27:51.206612 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.206597 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b99df94-3226-4a3f-9471-125bb7086943-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b99df94-3226-4a3f-9471-125bb7086943" (UID: "7b99df94-3226-4a3f-9471-125bb7086943"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:27:51.306273 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.306243 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b99df94-3226-4a3f-9471-125bb7086943-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:51.306273 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.306272 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b99df94-3226-4a3f-9471-125bb7086943-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:51.306273 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.306283 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9rkx\" (UniqueName: \"kubernetes.io/projected/7b99df94-3226-4a3f-9471-125bb7086943-kube-api-access-d9rkx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:51.652931 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.652894 2569 generic.go:356] "Generic (PLEG): container finished" podID="7b99df94-3226-4a3f-9471-125bb7086943" containerID="a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb" exitCode=0 Jul 21 14:27:51.653117 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.652943 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" Jul 21 14:27:51.653117 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.652979 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" event={"ID":"7b99df94-3226-4a3f-9471-125bb7086943","Type":"ContainerDied","Data":"a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb"} Jul 21 14:27:51.653117 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.653023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j" event={"ID":"7b99df94-3226-4a3f-9471-125bb7086943","Type":"ContainerDied","Data":"a3959c9041ee2d06a0b92f1a37e265e879793b358d90c8ab4a67cad2ab81a6f1"} Jul 21 14:27:51.653117 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.653041 2569 scope.go:117] "RemoveContainer" containerID="a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb" Jul 21 14:27:51.662251 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.662231 2569 scope.go:117] "RemoveContainer" containerID="a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb" Jul 21 14:27:51.662480 ip-10-0-129-45 kubenswrapper[2569]: E0721 14:27:51.662465 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb\": container with ID starting with a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb not found: ID does not exist" containerID="a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb" Jul 21 14:27:51.662533 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.662487 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb"} err="failed to get container status \"a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb\": rpc error: code = NotFound desc = could not find container \"a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb\": container with ID starting with a852cd3d7a37585b0521737ddee9487a6e1e286140814dc1efead1e98f1f78fb not found: ID does not exist" Jul 21 14:27:51.676222 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.675973 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j"] Jul 21 14:27:51.678621 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.678600 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-67be07/sub-test-model-67be07-kserve-55f445d885-qn92j"] Jul 21 14:27:51.705486 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.705453 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" podUID="1091bd22-b0c9-4407-a925-80dda8a34544" containerName="main" containerID="cri-o://b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f" gracePeriod=45 Jul 21 14:27:51.959853 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:51.959829 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:27:52.011065 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011035 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-model-cache\") pod \"1091bd22-b0c9-4407-a925-80dda8a34544\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " Jul 21 14:27:52.011238 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011106 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-tmp-dir\") pod \"1091bd22-b0c9-4407-a925-80dda8a34544\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " Jul 21 14:27:52.011238 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011142 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-home\") pod \"1091bd22-b0c9-4407-a925-80dda8a34544\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " Jul 21 14:27:52.011238 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011183 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf8r5\" (UniqueName: \"kubernetes.io/projected/1091bd22-b0c9-4407-a925-80dda8a34544-kube-api-access-vf8r5\") pod \"1091bd22-b0c9-4407-a925-80dda8a34544\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " Jul 21 14:27:52.011375 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011246 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-dshm\") pod \"1091bd22-b0c9-4407-a925-80dda8a34544\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " Jul 21 14:27:52.011375 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011302 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1091bd22-b0c9-4407-a925-80dda8a34544-tls-certs\") pod \"1091bd22-b0c9-4407-a925-80dda8a34544\" (UID: \"1091bd22-b0c9-4407-a925-80dda8a34544\") " Jul 21 14:27:52.011489 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011367 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-model-cache" (OuterVolumeSpecName: "model-cache") pod "1091bd22-b0c9-4407-a925-80dda8a34544" (UID: "1091bd22-b0c9-4407-a925-80dda8a34544"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:27:52.011489 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011382 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1091bd22-b0c9-4407-a925-80dda8a34544" (UID: "1091bd22-b0c9-4407-a925-80dda8a34544"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:27:52.011489 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011396 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-home" (OuterVolumeSpecName: "home") pod "1091bd22-b0c9-4407-a925-80dda8a34544" (UID: "1091bd22-b0c9-4407-a925-80dda8a34544"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:27:52.011763 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011739 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:52.011763 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011763 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:52.011936 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.011774 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:52.013532 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.013511 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-dshm" (OuterVolumeSpecName: "dshm") pod "1091bd22-b0c9-4407-a925-80dda8a34544" (UID: "1091bd22-b0c9-4407-a925-80dda8a34544"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:27:52.013734 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.013712 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1091bd22-b0c9-4407-a925-80dda8a34544-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1091bd22-b0c9-4407-a925-80dda8a34544" (UID: "1091bd22-b0c9-4407-a925-80dda8a34544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:27:52.013817 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.013733 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1091bd22-b0c9-4407-a925-80dda8a34544-kube-api-access-vf8r5" (OuterVolumeSpecName: "kube-api-access-vf8r5") pod "1091bd22-b0c9-4407-a925-80dda8a34544" (UID: "1091bd22-b0c9-4407-a925-80dda8a34544"). InnerVolumeSpecName "kube-api-access-vf8r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:27:52.112865 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.112829 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8r5\" (UniqueName: \"kubernetes.io/projected/1091bd22-b0c9-4407-a925-80dda8a34544-kube-api-access-vf8r5\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:52.112865 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.112861 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1091bd22-b0c9-4407-a925-80dda8a34544-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:52.112865 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.112872 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1091bd22-b0c9-4407-a925-80dda8a34544-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:27:52.659324 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.659290 2569 generic.go:356] "Generic (PLEG): container finished" podID="1091bd22-b0c9-4407-a925-80dda8a34544" containerID="b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f" exitCode=0 Jul 21 14:27:52.659494 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.659333 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" Jul 21 14:27:52.659494 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.659383 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" event={"ID":"1091bd22-b0c9-4407-a925-80dda8a34544","Type":"ContainerDied","Data":"b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f"} Jul 21 14:27:52.659494 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.659432 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq" event={"ID":"1091bd22-b0c9-4407-a925-80dda8a34544","Type":"ContainerDied","Data":"c92ad79cf565f8eaea37bd5f4c32212564686cea7a45cc68abc0a362f9e7d3f7"} Jul 21 14:27:52.659494 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.659451 2569 scope.go:117] "RemoveContainer" containerID="b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f" Jul 21 14:27:52.668446 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.668432 2569 scope.go:117] "RemoveContainer" containerID="b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f" Jul 21 14:27:52.668705 ip-10-0-129-45 kubenswrapper[2569]: E0721 14:27:52.668683 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f\": container with ID starting with b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f not found: ID does not exist" containerID="b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f" Jul 21 14:27:52.668763 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.668712 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f"} err="failed to get container status \"b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f\": rpc error: code = NotFound desc = could not find container \"b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f\": container with ID starting with b9d16fb82e9eed52ace26698f8ee8c05863118a87a8d71c4cac85305614d203f not found: ID does not exist" Jul 21 14:27:52.682306 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.682282 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq"] Jul 21 14:27:52.688357 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.688327 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-52967c/sub-test-model-52967c-kserve-5ff5c4bdf9-wxtrq"] Jul 21 14:27:52.795650 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.795619 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1091bd22-b0c9-4407-a925-80dda8a34544" path="/var/lib/kubelet/pods/1091bd22-b0c9-4407-a925-80dda8a34544/volumes" Jul 21 14:27:52.796040 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:27:52.796027 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b99df94-3226-4a3f-9471-125bb7086943" path="/var/lib/kubelet/pods/7b99df94-3226-4a3f-9471-125bb7086943/volumes" Jul 21 14:29:34.860003 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.859968 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140160148752816"] Jul 21 14:29:34.860625 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.860606 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1091bd22-b0c9-4407-a925-80dda8a34544" containerName="main" Jul 21 14:29:34.860726 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.860627 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1091bd22-b0c9-4407-a925-80dda8a34544" containerName="main" Jul 21 14:29:34.860726 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.860643 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b99df94-3226-4a3f-9471-125bb7086943" containerName="main" Jul 21 14:29:34.860726 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.860650 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b99df94-3226-4a3f-9471-125bb7086943" containerName="main" Jul 21 14:29:34.860877 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.860751 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1091bd22-b0c9-4407-a925-80dda8a34544" containerName="main" Jul 21 14:29:34.860877 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.860771 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b99df94-3226-4a3f-9471-125bb7086943" containerName="main" Jul 21 14:29:34.864101 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.864079 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:34.867215 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.867194 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8jrkq\"" Jul 21 14:29:34.869147 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.869128 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148752816"] Jul 21 14:29:34.914710 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:34.914677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xpv\" (UniqueName: \"kubernetes.io/projected/555e0380-cca7-401b-8e67-e32de5974340-kube-api-access-98xpv\") pod \"test-curl-7574-140160148752816\" (UID: \"555e0380-cca7-401b-8e67-e32de5974340\") " pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:35.016008 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:35.015969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xpv\" (UniqueName: \"kubernetes.io/projected/555e0380-cca7-401b-8e67-e32de5974340-kube-api-access-98xpv\") pod \"test-curl-7574-140160148752816\" (UID: \"555e0380-cca7-401b-8e67-e32de5974340\") " pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:35.025383 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:35.025355 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xpv\" (UniqueName: \"kubernetes.io/projected/555e0380-cca7-401b-8e67-e32de5974340-kube-api-access-98xpv\") pod \"test-curl-7574-140160148752816\" (UID: \"555e0380-cca7-401b-8e67-e32de5974340\") " pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:35.174779 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:35.174690 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:35.304135 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:35.304110 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148752816"] Jul 21 14:29:35.306528 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:29:35.306495 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod555e0380_cca7_401b_8e67_e32de5974340.slice/crio-3b9bc9c464d4d4a2cd1b8bc82d3f4cd04af369bc5435f3662e9922a00c5da2f4 WatchSource:0}: Error finding container 3b9bc9c464d4d4a2cd1b8bc82d3f4cd04af369bc5435f3662e9922a00c5da2f4: Status 404 returned error can't find the container with id 3b9bc9c464d4d4a2cd1b8bc82d3f4cd04af369bc5435f3662e9922a00c5da2f4 Jul 21 14:29:35.308260 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:35.308238 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 14:29:36.175219 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:36.175188 2569 generic.go:356] "Generic (PLEG): container finished" podID="555e0380-cca7-401b-8e67-e32de5974340" containerID="155d0078314b9c7f78955620b193d4ed5495f7d156060e058904635f04651f10" exitCode=0 Jul 21 14:29:36.175626 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:36.175272 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148752816" event={"ID":"555e0380-cca7-401b-8e67-e32de5974340","Type":"ContainerDied","Data":"155d0078314b9c7f78955620b193d4ed5495f7d156060e058904635f04651f10"} Jul 21 14:29:36.175626 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:36.175313 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148752816" event={"ID":"555e0380-cca7-401b-8e67-e32de5974340","Type":"ContainerStarted","Data":"3b9bc9c464d4d4a2cd1b8bc82d3f4cd04af369bc5435f3662e9922a00c5da2f4"} Jul 21 14:29:37.312218 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.312189 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:37.337036 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.337004 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98xpv\" (UniqueName: \"kubernetes.io/projected/555e0380-cca7-401b-8e67-e32de5974340-kube-api-access-98xpv\") pod \"555e0380-cca7-401b-8e67-e32de5974340\" (UID: \"555e0380-cca7-401b-8e67-e32de5974340\") " Jul 21 14:29:37.339174 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.339145 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555e0380-cca7-401b-8e67-e32de5974340-kube-api-access-98xpv" (OuterVolumeSpecName: "kube-api-access-98xpv") pod "555e0380-cca7-401b-8e67-e32de5974340" (UID: "555e0380-cca7-401b-8e67-e32de5974340"). InnerVolumeSpecName "kube-api-access-98xpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:29:37.350450 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.350429 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140160148752816_555e0380-cca7-401b-8e67-e32de5974340/test-curl-7574-140160148752816/0.log" Jul 21 14:29:37.379373 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.379341 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140160148752816"] Jul 21 14:29:37.381142 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.381121 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140160148752816"] Jul 21 14:29:37.438256 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.438173 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xpv\" (UniqueName: \"kubernetes.io/projected/555e0380-cca7-401b-8e67-e32de5974340-kube-api-access-98xpv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:29:37.502787 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.502756 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140160148743344"] Jul 21 14:29:37.503160 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.503146 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555e0380-cca7-401b-8e67-e32de5974340" containerName="test-curl-7574-140160148752816" Jul 21 14:29:37.503204 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.503162 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="555e0380-cca7-401b-8e67-e32de5974340" containerName="test-curl-7574-140160148752816" Jul 21 14:29:37.503238 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.503230 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="555e0380-cca7-401b-8e67-e32de5974340" containerName="test-curl-7574-140160148752816" Jul 21 14:29:37.506220 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.506204 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148743344" Jul 21 14:29:37.512906 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.512883 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148743344"] Jul 21 14:29:37.538696 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.538667 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-287qm\" (UniqueName: \"kubernetes.io/projected/06f238d9-67c0-4e19-aadf-c62e89cd93de-kube-api-access-287qm\") pod \"test-curl-7574-140160148743344\" (UID: \"06f238d9-67c0-4e19-aadf-c62e89cd93de\") " pod="opendatahub/test-curl-7574-140160148743344" Jul 21 14:29:37.640162 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.640127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-287qm\" (UniqueName: \"kubernetes.io/projected/06f238d9-67c0-4e19-aadf-c62e89cd93de-kube-api-access-287qm\") pod \"test-curl-7574-140160148743344\" (UID: \"06f238d9-67c0-4e19-aadf-c62e89cd93de\") " pod="opendatahub/test-curl-7574-140160148743344" Jul 21 14:29:37.648870 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.648839 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-287qm\" (UniqueName: \"kubernetes.io/projected/06f238d9-67c0-4e19-aadf-c62e89cd93de-kube-api-access-287qm\") pod \"test-curl-7574-140160148743344\" (UID: \"06f238d9-67c0-4e19-aadf-c62e89cd93de\") " pod="opendatahub/test-curl-7574-140160148743344" Jul 21 14:29:37.816469 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.816436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148743344" Jul 21 14:29:37.948954 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:37.948927 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148743344"] Jul 21 14:29:37.951311 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:29:37.951281 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f238d9_67c0_4e19_aadf_c62e89cd93de.slice/crio-cda101ac787f4059c87078ac63d4957e5c7bc2577d4e084bdac2d6cd6734db26 WatchSource:0}: Error finding container cda101ac787f4059c87078ac63d4957e5c7bc2577d4e084bdac2d6cd6734db26: Status 404 returned error can't find the container with id cda101ac787f4059c87078ac63d4957e5c7bc2577d4e084bdac2d6cd6734db26 Jul 21 14:29:38.186210 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:38.186124 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148743344" event={"ID":"06f238d9-67c0-4e19-aadf-c62e89cd93de","Type":"ContainerStarted","Data":"cda101ac787f4059c87078ac63d4957e5c7bc2577d4e084bdac2d6cd6734db26"} Jul 21 14:29:38.187256 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:38.187237 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:38.187256 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:38.187243 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b9bc9c464d4d4a2cd1b8bc82d3f4cd04af369bc5435f3662e9922a00c5da2f4" Jul 21 14:29:38.198710 ip-10-0-129-45 kubenswrapper[2569]: E0721 14:29:38.198684 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7574-140160148752816\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="555e0380-cca7-401b-8e67-e32de5974340" pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:38.794746 ip-10-0-129-45 kubenswrapper[2569]: E0721 14:29:38.794706 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7574-140160148752816\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="555e0380-cca7-401b-8e67-e32de5974340" pod="opendatahub/test-curl-7574-140160148752816" Jul 21 14:29:38.796179 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:38.796158 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="555e0380-cca7-401b-8e67-e32de5974340" path="/var/lib/kubelet/pods/555e0380-cca7-401b-8e67-e32de5974340/volumes" Jul 21 14:29:39.193891 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:39.193801 2569 generic.go:356] "Generic (PLEG): container finished" podID="06f238d9-67c0-4e19-aadf-c62e89cd93de" containerID="8b9995a1027cf717f0d2fdde6cf2566ef1b85c37217dee7cacb1ebe1f3d02837" exitCode=0 Jul 21 14:29:39.194033 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:39.193895 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148743344" event={"ID":"06f238d9-67c0-4e19-aadf-c62e89cd93de","Type":"ContainerDied","Data":"8b9995a1027cf717f0d2fdde6cf2566ef1b85c37217dee7cacb1ebe1f3d02837"} Jul 21 14:29:40.331520 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.331484 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148743344" Jul 21 14:29:40.355013 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.354989 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140160148743344_06f238d9-67c0-4e19-aadf-c62e89cd93de/test-curl-7574-140160148743344/0.log" Jul 21 14:29:40.366253 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.366221 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-287qm\" (UniqueName: \"kubernetes.io/projected/06f238d9-67c0-4e19-aadf-c62e89cd93de-kube-api-access-287qm\") pod \"06f238d9-67c0-4e19-aadf-c62e89cd93de\" (UID: \"06f238d9-67c0-4e19-aadf-c62e89cd93de\") " Jul 21 14:29:40.368251 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.368221 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f238d9-67c0-4e19-aadf-c62e89cd93de-kube-api-access-287qm" (OuterVolumeSpecName: "kube-api-access-287qm") pod "06f238d9-67c0-4e19-aadf-c62e89cd93de" (UID: "06f238d9-67c0-4e19-aadf-c62e89cd93de"). InnerVolumeSpecName "kube-api-access-287qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:29:40.384794 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.384770 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140160148743344"] Jul 21 14:29:40.391056 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.391038 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140160148743344"] Jul 21 14:29:40.468137 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.468060 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-287qm\" (UniqueName: \"kubernetes.io/projected/06f238d9-67c0-4e19-aadf-c62e89cd93de-kube-api-access-287qm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:29:40.552397 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.552361 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:40.552781 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.552767 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f238d9-67c0-4e19-aadf-c62e89cd93de" containerName="test-curl-7574-140160148743344" Jul 21 14:29:40.552838 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.552784 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f238d9-67c0-4e19-aadf-c62e89cd93de" containerName="test-curl-7574-140160148743344" Jul 21 14:29:40.552878 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.552870 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f238d9-67c0-4e19-aadf-c62e89cd93de" containerName="test-curl-7574-140160148743344" Jul 21 14:29:40.555850 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.555833 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:40.560984 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.560960 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:40.670488 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.670447 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj7xv\" (UniqueName: \"kubernetes.io/projected/7e21c8d9-dad5-4ff5-9437-c5c52fcadd97-kube-api-access-kj7xv\") pod \"test-curl-7574-140160148735296\" (UID: \"7e21c8d9-dad5-4ff5-9437-c5c52fcadd97\") " pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:40.771442 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.771405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj7xv\" (UniqueName: \"kubernetes.io/projected/7e21c8d9-dad5-4ff5-9437-c5c52fcadd97-kube-api-access-kj7xv\") pod \"test-curl-7574-140160148735296\" (UID: \"7e21c8d9-dad5-4ff5-9437-c5c52fcadd97\") " pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:40.780079 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.780057 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj7xv\" (UniqueName: \"kubernetes.io/projected/7e21c8d9-dad5-4ff5-9437-c5c52fcadd97-kube-api-access-kj7xv\") pod \"test-curl-7574-140160148735296\" (UID: \"7e21c8d9-dad5-4ff5-9437-c5c52fcadd97\") " pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:40.796077 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.796048 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f238d9-67c0-4e19-aadf-c62e89cd93de" path="/var/lib/kubelet/pods/06f238d9-67c0-4e19-aadf-c62e89cd93de/volumes" Jul 21 14:29:40.866401 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.866372 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:40.998956 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:40.998932 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:41.000839 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:29:41.000813 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e21c8d9_dad5_4ff5_9437_c5c52fcadd97.slice/crio-b54cf5482071a5359f71f43ce5da99a1dedb5baa16d0e82d4cf2eac925f19f1a WatchSource:0}: Error finding container b54cf5482071a5359f71f43ce5da99a1dedb5baa16d0e82d4cf2eac925f19f1a: Status 404 returned error can't find the container with id b54cf5482071a5359f71f43ce5da99a1dedb5baa16d0e82d4cf2eac925f19f1a Jul 21 14:29:41.204342 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:41.204304 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148743344" Jul 21 14:29:41.204342 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:41.204349 2569 scope.go:117] "RemoveContainer" containerID="8b9995a1027cf717f0d2fdde6cf2566ef1b85c37217dee7cacb1ebe1f3d02837" Jul 21 14:29:41.205837 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:41.205666 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148735296" event={"ID":"7e21c8d9-dad5-4ff5-9437-c5c52fcadd97","Type":"ContainerStarted","Data":"b54cf5482071a5359f71f43ce5da99a1dedb5baa16d0e82d4cf2eac925f19f1a"} Jul 21 14:29:42.213511 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:42.213474 2569 generic.go:356] "Generic (PLEG): container finished" podID="7e21c8d9-dad5-4ff5-9437-c5c52fcadd97" containerID="89dc9390f782858b286367d3fd15fc4f8011dad14140780c31c9ead488ad15b8" exitCode=0 Jul 21 14:29:42.213915 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:42.213556 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148735296" event={"ID":"7e21c8d9-dad5-4ff5-9437-c5c52fcadd97","Type":"ContainerDied","Data":"89dc9390f782858b286367d3fd15fc4f8011dad14140780c31c9ead488ad15b8"} Jul 21 14:29:43.355266 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.355242 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:43.374990 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.374960 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140160148735296_7e21c8d9-dad5-4ff5-9437-c5c52fcadd97/test-curl-7574-140160148735296/0.log" Jul 21 14:29:43.397206 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.397176 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:43.399217 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.399200 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj7xv\" (UniqueName: \"kubernetes.io/projected/7e21c8d9-dad5-4ff5-9437-c5c52fcadd97-kube-api-access-kj7xv\") pod \"7e21c8d9-dad5-4ff5-9437-c5c52fcadd97\" (UID: \"7e21c8d9-dad5-4ff5-9437-c5c52fcadd97\") " Jul 21 14:29:43.401531 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.401505 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e21c8d9-dad5-4ff5-9437-c5c52fcadd97-kube-api-access-kj7xv" (OuterVolumeSpecName: "kube-api-access-kj7xv") pod "7e21c8d9-dad5-4ff5-9437-c5c52fcadd97" (UID: "7e21c8d9-dad5-4ff5-9437-c5c52fcadd97"). InnerVolumeSpecName "kube-api-access-kj7xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:29:43.402457 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.402436 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:43.500668 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.500639 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj7xv\" (UniqueName: \"kubernetes.io/projected/7e21c8d9-dad5-4ff5-9437-c5c52fcadd97-kube-api-access-kj7xv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:29:43.518000 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.517972 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:43.518401 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.518387 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e21c8d9-dad5-4ff5-9437-c5c52fcadd97" containerName="test-curl-7574-140160148735296" Jul 21 14:29:43.518445 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.518403 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e21c8d9-dad5-4ff5-9437-c5c52fcadd97" containerName="test-curl-7574-140160148735296" Jul 21 14:29:43.518497 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.518488 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e21c8d9-dad5-4ff5-9437-c5c52fcadd97" containerName="test-curl-7574-140160148735296" Jul 21 14:29:43.521511 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.521496 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:43.528965 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.528941 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:43.601889 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.601862 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9whx\" (UniqueName: \"kubernetes.io/projected/20e76b37-8e82-44f5-91ca-8e2018b050e6-kube-api-access-l9whx\") pod \"test-curl-7574-140160148735296\" (UID: \"20e76b37-8e82-44f5-91ca-8e2018b050e6\") " pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:43.703280 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.703245 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9whx\" (UniqueName: \"kubernetes.io/projected/20e76b37-8e82-44f5-91ca-8e2018b050e6-kube-api-access-l9whx\") pod \"test-curl-7574-140160148735296\" (UID: \"20e76b37-8e82-44f5-91ca-8e2018b050e6\") " pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:43.712598 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.712571 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9whx\" (UniqueName: \"kubernetes.io/projected/20e76b37-8e82-44f5-91ca-8e2018b050e6-kube-api-access-l9whx\") pod \"test-curl-7574-140160148735296\" (UID: \"20e76b37-8e82-44f5-91ca-8e2018b050e6\") " pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:43.832012 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.831941 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:43.966322 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:43.966297 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:43.969023 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:29:43.968996 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20e76b37_8e82_44f5_91ca_8e2018b050e6.slice/crio-7d994bbc3e36d51d9ccaaf0921e8af5c9fee08e15aa9158014e957dd51fa4c4f WatchSource:0}: Error finding container 7d994bbc3e36d51d9ccaaf0921e8af5c9fee08e15aa9158014e957dd51fa4c4f: Status 404 returned error can't find the container with id 7d994bbc3e36d51d9ccaaf0921e8af5c9fee08e15aa9158014e957dd51fa4c4f Jul 21 14:29:44.224082 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:44.223999 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:44.224082 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:44.224041 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b54cf5482071a5359f71f43ce5da99a1dedb5baa16d0e82d4cf2eac925f19f1a" Jul 21 14:29:44.225289 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:44.225260 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148735296" event={"ID":"20e76b37-8e82-44f5-91ca-8e2018b050e6","Type":"ContainerStarted","Data":"7d994bbc3e36d51d9ccaaf0921e8af5c9fee08e15aa9158014e957dd51fa4c4f"} Jul 21 14:29:44.236234 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:44.236205 2569 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7574-140160148735296" oldPodUID="7e21c8d9-dad5-4ff5-9437-c5c52fcadd97" podUID="20e76b37-8e82-44f5-91ca-8e2018b050e6" Jul 21 14:29:44.796513 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:44.796476 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e21c8d9-dad5-4ff5-9437-c5c52fcadd97" path="/var/lib/kubelet/pods/7e21c8d9-dad5-4ff5-9437-c5c52fcadd97/volumes" Jul 21 14:29:45.231653 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:45.231619 2569 generic.go:356] "Generic (PLEG): container finished" podID="20e76b37-8e82-44f5-91ca-8e2018b050e6" containerID="6654169aa02a41ebb584d2f479ec77e815352c9813da4235c559480e0c33ab26" exitCode=0 Jul 21 14:29:45.231818 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:45.231709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148735296" event={"ID":"20e76b37-8e82-44f5-91ca-8e2018b050e6","Type":"ContainerDied","Data":"6654169aa02a41ebb584d2f479ec77e815352c9813da4235c559480e0c33ab26"} Jul 21 14:29:46.373957 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.373931 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:46.395770 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.395737 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140160148735296_20e76b37-8e82-44f5-91ca-8e2018b050e6/test-curl-7574-140160148735296/0.log" Jul 21 14:29:46.419510 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.419481 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:46.422994 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.422971 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140160148735296"] Jul 21 14:29:46.429329 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.429313 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9whx\" (UniqueName: \"kubernetes.io/projected/20e76b37-8e82-44f5-91ca-8e2018b050e6-kube-api-access-l9whx\") pod \"20e76b37-8e82-44f5-91ca-8e2018b050e6\" (UID: \"20e76b37-8e82-44f5-91ca-8e2018b050e6\") " Jul 21 14:29:46.431424 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.431400 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e76b37-8e82-44f5-91ca-8e2018b050e6-kube-api-access-l9whx" (OuterVolumeSpecName: "kube-api-access-l9whx") pod "20e76b37-8e82-44f5-91ca-8e2018b050e6" (UID: "20e76b37-8e82-44f5-91ca-8e2018b050e6"). InnerVolumeSpecName "kube-api-access-l9whx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:29:46.530094 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.530026 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9whx\" (UniqueName: \"kubernetes.io/projected/20e76b37-8e82-44f5-91ca-8e2018b050e6-kube-api-access-l9whx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:29:46.545233 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.545205 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:46.545614 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.545601 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e76b37-8e82-44f5-91ca-8e2018b050e6" containerName="test-curl-7574-140160148735296" Jul 21 14:29:46.545661 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.545616 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e76b37-8e82-44f5-91ca-8e2018b050e6" containerName="test-curl-7574-140160148735296" Jul 21 14:29:46.545720 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.545711 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e76b37-8e82-44f5-91ca-8e2018b050e6" containerName="test-curl-7574-140160148735296" Jul 21 14:29:46.548724 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.548709 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:46.555792 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.555753 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:46.631010 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.630979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwg9w\" (UniqueName: \"kubernetes.io/projected/113ef2a6-ea28-4f90-96ee-252d6f0c0bce-kube-api-access-mwg9w\") pod \"test-curl-7574-140160148736304\" (UID: \"113ef2a6-ea28-4f90-96ee-252d6f0c0bce\") " pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:46.731720 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.731681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwg9w\" (UniqueName: \"kubernetes.io/projected/113ef2a6-ea28-4f90-96ee-252d6f0c0bce-kube-api-access-mwg9w\") pod \"test-curl-7574-140160148736304\" (UID: \"113ef2a6-ea28-4f90-96ee-252d6f0c0bce\") " pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:46.741472 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.741433 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwg9w\" (UniqueName: \"kubernetes.io/projected/113ef2a6-ea28-4f90-96ee-252d6f0c0bce-kube-api-access-mwg9w\") pod \"test-curl-7574-140160148736304\" (UID: \"113ef2a6-ea28-4f90-96ee-252d6f0c0bce\") " pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:46.795877 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.795793 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20e76b37-8e82-44f5-91ca-8e2018b050e6" path="/var/lib/kubelet/pods/20e76b37-8e82-44f5-91ca-8e2018b050e6/volumes" Jul 21 14:29:46.858242 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.858206 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:46.994138 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:46.994112 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:46.996217 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:29:46.996190 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod113ef2a6_ea28_4f90_96ee_252d6f0c0bce.slice/crio-411d3e8d61875d32120e85201025ceb89870b9995efe212e721c61d2b5cb1a45 WatchSource:0}: Error finding container 411d3e8d61875d32120e85201025ceb89870b9995efe212e721c61d2b5cb1a45: Status 404 returned error can't find the container with id 411d3e8d61875d32120e85201025ceb89870b9995efe212e721c61d2b5cb1a45 Jul 21 14:29:47.244172 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:47.244137 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148736304" event={"ID":"113ef2a6-ea28-4f90-96ee-252d6f0c0bce","Type":"ContainerStarted","Data":"411d3e8d61875d32120e85201025ceb89870b9995efe212e721c61d2b5cb1a45"} Jul 21 14:29:47.245330 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:47.245307 2569 scope.go:117] "RemoveContainer" containerID="6654169aa02a41ebb584d2f479ec77e815352c9813da4235c559480e0c33ab26" Jul 21 14:29:47.245464 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:47.245346 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148735296" Jul 21 14:29:48.251190 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:48.251155 2569 generic.go:356] "Generic (PLEG): container finished" podID="113ef2a6-ea28-4f90-96ee-252d6f0c0bce" containerID="f163763b3cae141d27fc547c31f3056e606cd2433254ef553e4b2929ba5e8f6b" exitCode=0 Jul 21 14:29:48.251639 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:48.251243 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148736304" event={"ID":"113ef2a6-ea28-4f90-96ee-252d6f0c0bce","Type":"ContainerDied","Data":"f163763b3cae141d27fc547c31f3056e606cd2433254ef553e4b2929ba5e8f6b"} Jul 21 14:29:49.391400 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.391374 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:49.412652 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.412627 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140160148736304_113ef2a6-ea28-4f90-96ee-252d6f0c0bce/test-curl-7574-140160148736304/0.log" Jul 21 14:29:49.439766 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.439733 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:49.445745 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.445721 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:49.459773 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.459753 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwg9w\" (UniqueName: \"kubernetes.io/projected/113ef2a6-ea28-4f90-96ee-252d6f0c0bce-kube-api-access-mwg9w\") pod \"113ef2a6-ea28-4f90-96ee-252d6f0c0bce\" (UID: \"113ef2a6-ea28-4f90-96ee-252d6f0c0bce\") " Jul 21 14:29:49.461809 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.461786 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113ef2a6-ea28-4f90-96ee-252d6f0c0bce-kube-api-access-mwg9w" (OuterVolumeSpecName: "kube-api-access-mwg9w") pod "113ef2a6-ea28-4f90-96ee-252d6f0c0bce" (UID: "113ef2a6-ea28-4f90-96ee-252d6f0c0bce"). InnerVolumeSpecName "kube-api-access-mwg9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:29:49.561208 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.561122 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwg9w\" (UniqueName: \"kubernetes.io/projected/113ef2a6-ea28-4f90-96ee-252d6f0c0bce-kube-api-access-mwg9w\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:29:49.561737 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.561716 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:49.562127 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.562115 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113ef2a6-ea28-4f90-96ee-252d6f0c0bce" containerName="test-curl-7574-140160148736304" Jul 21 14:29:49.562174 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.562129 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="113ef2a6-ea28-4f90-96ee-252d6f0c0bce" containerName="test-curl-7574-140160148736304" Jul 21 14:29:49.562209 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.562204 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="113ef2a6-ea28-4f90-96ee-252d6f0c0bce" containerName="test-curl-7574-140160148736304" Jul 21 14:29:49.565226 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.565210 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:49.569978 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.569951 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:49.662436 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.662399 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhrp\" (UniqueName: \"kubernetes.io/projected/02477201-c7a8-46b8-900a-556f80412529-kube-api-access-rmhrp\") pod \"test-curl-7574-140160148736304\" (UID: \"02477201-c7a8-46b8-900a-556f80412529\") " pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:49.763655 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.763613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhrp\" (UniqueName: \"kubernetes.io/projected/02477201-c7a8-46b8-900a-556f80412529-kube-api-access-rmhrp\") pod \"test-curl-7574-140160148736304\" (UID: \"02477201-c7a8-46b8-900a-556f80412529\") " pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:49.772935 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.772900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhrp\" (UniqueName: \"kubernetes.io/projected/02477201-c7a8-46b8-900a-556f80412529-kube-api-access-rmhrp\") pod \"test-curl-7574-140160148736304\" (UID: \"02477201-c7a8-46b8-900a-556f80412529\") " pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:49.876070 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:49.875990 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:50.010663 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:50.010636 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:50.012410 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:29:50.012383 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02477201_c7a8_46b8_900a_556f80412529.slice/crio-2a289bb1f8ec57697df6ae0b087341ed06955ee4d71e1222b3fae47ff9bfb59d WatchSource:0}: Error finding container 2a289bb1f8ec57697df6ae0b087341ed06955ee4d71e1222b3fae47ff9bfb59d: Status 404 returned error can't find the container with id 2a289bb1f8ec57697df6ae0b087341ed06955ee4d71e1222b3fae47ff9bfb59d Jul 21 14:29:50.264768 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:50.264729 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:50.264938 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:50.264728 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="411d3e8d61875d32120e85201025ceb89870b9995efe212e721c61d2b5cb1a45" Jul 21 14:29:50.265842 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:50.265819 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148736304" event={"ID":"02477201-c7a8-46b8-900a-556f80412529","Type":"ContainerStarted","Data":"2a289bb1f8ec57697df6ae0b087341ed06955ee4d71e1222b3fae47ff9bfb59d"} Jul 21 14:29:50.276409 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:50.276384 2569 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7574-140160148736304" oldPodUID="113ef2a6-ea28-4f90-96ee-252d6f0c0bce" podUID="02477201-c7a8-46b8-900a-556f80412529" Jul 21 14:29:50.801972 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:50.801938 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113ef2a6-ea28-4f90-96ee-252d6f0c0bce" path="/var/lib/kubelet/pods/113ef2a6-ea28-4f90-96ee-252d6f0c0bce/volumes" Jul 21 14:29:51.272020 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:51.271991 2569 generic.go:356] "Generic (PLEG): container finished" podID="02477201-c7a8-46b8-900a-556f80412529" containerID="7d263a068d21820d0b9b26db95530f502ff8592db22e4432bd298f7a854a8088" exitCode=0 Jul 21 14:29:51.272209 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:51.272073 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140160148736304" event={"ID":"02477201-c7a8-46b8-900a-556f80412529","Type":"ContainerDied","Data":"7d263a068d21820d0b9b26db95530f502ff8592db22e4432bd298f7a854a8088"} Jul 21 14:29:52.413970 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:52.413945 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:52.434173 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:52.434143 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140160148736304_02477201-c7a8-46b8-900a-556f80412529/test-curl-7574-140160148736304/0.log" Jul 21 14:29:52.457846 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:52.457822 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:52.464185 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:52.464163 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140160148736304"] Jul 21 14:29:52.493630 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:52.493605 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmhrp\" (UniqueName: \"kubernetes.io/projected/02477201-c7a8-46b8-900a-556f80412529-kube-api-access-rmhrp\") pod \"02477201-c7a8-46b8-900a-556f80412529\" (UID: \"02477201-c7a8-46b8-900a-556f80412529\") " Jul 21 14:29:52.495762 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:52.495737 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02477201-c7a8-46b8-900a-556f80412529-kube-api-access-rmhrp" (OuterVolumeSpecName: "kube-api-access-rmhrp") pod "02477201-c7a8-46b8-900a-556f80412529" (UID: "02477201-c7a8-46b8-900a-556f80412529"). InnerVolumeSpecName "kube-api-access-rmhrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:29:52.594580 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:52.594472 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhrp\" (UniqueName: \"kubernetes.io/projected/02477201-c7a8-46b8-900a-556f80412529-kube-api-access-rmhrp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\"" Jul 21 14:29:52.795776 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:52.795745 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02477201-c7a8-46b8-900a-556f80412529" path="/var/lib/kubelet/pods/02477201-c7a8-46b8-900a-556f80412529/volumes" Jul 21 14:29:53.283082 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:53.283051 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140160148736304" Jul 21 14:29:53.283082 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:29:53.283077 2569 scope.go:117] "RemoveContainer" containerID="7d263a068d21820d0b9b26db95530f502ff8592db22e4432bd298f7a854a8088" Jul 21 14:30:44.037568 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:44.037524 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b6bd44f67-rh5qr_64f132c4-bccf-47f2-9c9e-4205e33f5a83/manager/0.log" Jul 21 14:30:44.387585 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:44.387451 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d98d8798d-ccn85_fb6bd71d-faad-4dda-8283-b2887028bbe5/copy-manifests/0.log" Jul 21 14:30:44.399755 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:44.399727 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d98d8798d-ccn85_fb6bd71d-faad-4dda-8283-b2887028bbe5/manager/0.log" Jul 21 14:30:44.983941 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:44.983911 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-745854dcc7-jnkcw_18eb1fed-0663-4034-8a5e-59ea83a6469e/maas-api/0.log" Jul 21 14:30:45.426586 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:45.426481 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-nk8k9_c5ee636e-d59b-4353-9f9c-944f80a5cf47/postgres/0.log" Jul 21 14:30:47.057733 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:47.057701 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-fmkhq_3c4f19b8-8a85-45eb-bfaa-3df67c186c01/kuadrant-console-plugin/0.log" Jul 21 14:30:48.087512 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:48.087488 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55c6b9f88d-9hq54_4119d337-191a-4129-8626-ed2505a0854e/kube-auth-proxy/0.log" Jul 21 14:30:48.439063 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:48.438983 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-prk4n_e9761259-3cef-4858-824d-ba2157670e1f/payload-pre-processing/0.log" Jul 21 14:30:49.006745 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:49.006712 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-dgfv4_895ba857-bb3b-4190-9cd2-f7771c3fda69/main/0.log" Jul 21 14:30:49.124559 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:49.124514 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-qksgm_38dd0609-7d6a-4195-a32a-86b006a5643c/main/0.log" Jul 21 14:30:49.351769 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:49.351681 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxrfvv_0b70798b-4d10-4051-b63c-0c6d17cbc6b6/main/0.log" Jul 21 14:30:59.167215 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:59.167181 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ztpvd_79c9e89d-d287-46a8-b4fe-eeb2e8a31223/global-pull-secret-syncer/0.log" Jul 21 14:30:59.262012 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:59.261986 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ktcmb_f018ddca-f4a1-444c-b602-f0a34a3f3269/konnectivity-agent/0.log" Jul 21 14:30:59.346240 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:30:59.346212 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-45.ec2.internal_3b9b4eb6aa309c78e5ae7c65fd70512d/haproxy/0.log" Jul 21 14:31:05.563491 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:05.563463 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-fmkhq_3c4f19b8-8a85-45eb-bfaa-3df67c186c01/kuadrant-console-plugin/0.log" Jul 21 14:31:07.724466 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:07.724434 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69/alertmanager/0.log" Jul 21 14:31:07.759185 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:07.759159 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69/config-reloader/0.log" Jul 21 14:31:07.784043 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:07.784021 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69/kube-rbac-proxy-web/0.log" Jul 21 14:31:07.809471 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:07.809450 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69/kube-rbac-proxy/0.log" Jul 21 14:31:07.843244 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:07.843219 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69/kube-rbac-proxy-metric/0.log" Jul 21 14:31:07.873739 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:07.873717 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69/prom-label-proxy/0.log" Jul 21 14:31:07.899593 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:07.899566 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d0e8ee1-a2df-4516-bf4f-f0d08c6fed69/init-config-reloader/0.log" Jul 21 14:31:07.990005 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:07.989969 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-g896z_1e67cdf6-b024-4463-879e-b95846974683/kube-state-metrics/0.log" Jul 21 14:31:08.012303 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.012278 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-g896z_1e67cdf6-b024-4463-879e-b95846974683/kube-rbac-proxy-main/0.log" Jul 21 14:31:08.044963 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.044941 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-g896z_1e67cdf6-b024-4463-879e-b95846974683/kube-rbac-proxy-self/0.log" Jul 21 14:31:08.077333 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.077304 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b857c966-2pwpf_2b5027e9-e351-47a0-8e33-e8184f4ebef0/metrics-server/0.log" Jul 21 14:31:08.111703 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.111678 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-682qm_e67d0ca1-8ff4-40e6-baa1-fd307def1ff9/monitoring-plugin/0.log" Jul 21 14:31:08.313509 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.313436 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xgrdw_9063bc3f-f3cb-4946-9841-b47b6503b7b4/node-exporter/0.log" Jul 21 14:31:08.337159 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.337131 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xgrdw_9063bc3f-f3cb-4946-9841-b47b6503b7b4/kube-rbac-proxy/0.log" Jul 21 14:31:08.360580 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.360534 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xgrdw_9063bc3f-f3cb-4946-9841-b47b6503b7b4/init-textfile/0.log" Jul 21 14:31:08.389290 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.389253 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-dtcsn_ba604b91-8658-4512-93f3-724667407ea1/kube-rbac-proxy-main/0.log" Jul 21 14:31:08.414489 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.414466 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-dtcsn_ba604b91-8658-4512-93f3-724667407ea1/kube-rbac-proxy-self/0.log" Jul 21 14:31:08.438194 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.438170 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-dtcsn_ba604b91-8658-4512-93f3-724667407ea1/openshift-state-metrics/0.log" Jul 21 14:31:08.503187 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.503161 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1ca33b6-5d9c-4cda-84f6-0079874d8b03/prometheus/0.log" Jul 21 14:31:08.542664 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.542638 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1ca33b6-5d9c-4cda-84f6-0079874d8b03/config-reloader/0.log" Jul 21 14:31:08.589462 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.589401 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1ca33b6-5d9c-4cda-84f6-0079874d8b03/thanos-sidecar/0.log" Jul 21 14:31:08.616139 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.616110 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1ca33b6-5d9c-4cda-84f6-0079874d8b03/kube-rbac-proxy-web/0.log" Jul 21 14:31:08.642690 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.642669 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1ca33b6-5d9c-4cda-84f6-0079874d8b03/kube-rbac-proxy/0.log" Jul 21 14:31:08.667749 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.667724 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1ca33b6-5d9c-4cda-84f6-0079874d8b03/kube-rbac-proxy-thanos/0.log" Jul 21 14:31:08.693733 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.693707 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1ca33b6-5d9c-4cda-84f6-0079874d8b03/init-config-reloader/0.log" Jul 21 14:31:08.726371 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.726340 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-ngt84_b91a43c7-5d49-493c-b568-1e96dca4340d/prometheus-operator/0.log" Jul 21 14:31:08.750853 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.750832 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-ngt84_b91a43c7-5d49-493c-b568-1e96dca4340d/kube-rbac-proxy/0.log" Jul 21 14:31:08.777729 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.777704 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-frjlj_c20e3db2-ed3c-4118-9171-90aae7440577/prometheus-operator-admission-webhook/0.log" Jul 21 14:31:08.815873 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.815846 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77dc48c6b5-8f6bs_3042006e-8243-49e9-a4cb-311457042576/telemeter-client/0.log" Jul 21 14:31:08.839039 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.839015 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77dc48c6b5-8f6bs_3042006e-8243-49e9-a4cb-311457042576/reload/0.log" Jul 21 14:31:08.861940 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.861872 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77dc48c6b5-8f6bs_3042006e-8243-49e9-a4cb-311457042576/kube-rbac-proxy/0.log" Jul 21 14:31:08.895208 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.895179 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5656dbdcdc-njbbd_20c56c13-3380-4a02-9d22-7f85f7ec346c/thanos-query/0.log" Jul 21 14:31:08.917971 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.917946 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5656dbdcdc-njbbd_20c56c13-3380-4a02-9d22-7f85f7ec346c/kube-rbac-proxy-web/0.log" Jul 21 14:31:08.944274 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.944247 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5656dbdcdc-njbbd_20c56c13-3380-4a02-9d22-7f85f7ec346c/kube-rbac-proxy/0.log" Jul 21 14:31:08.972031 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.972007 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5656dbdcdc-njbbd_20c56c13-3380-4a02-9d22-7f85f7ec346c/prom-label-proxy/0.log" Jul 21 14:31:08.997103 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:08.997081 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5656dbdcdc-njbbd_20c56c13-3380-4a02-9d22-7f85f7ec346c/kube-rbac-proxy-rules/0.log" Jul 21 14:31:09.023972 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:09.023945 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5656dbdcdc-njbbd_20c56c13-3380-4a02-9d22-7f85f7ec346c/kube-rbac-proxy-metrics/0.log" Jul 21 14:31:10.015980 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.015938 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p"] Jul 21 14:31:10.016394 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.016378 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02477201-c7a8-46b8-900a-556f80412529" containerName="test-curl-7574-140160148736304" Jul 21 14:31:10.016394 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.016394 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02477201-c7a8-46b8-900a-556f80412529" containerName="test-curl-7574-140160148736304" Jul 21 14:31:10.016482 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.016460 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="02477201-c7a8-46b8-900a-556f80412529" containerName="test-curl-7574-140160148736304" Jul 21 14:31:10.019715 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.019697 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.022438 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.022416 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fv4cl\"/\"default-dockercfg-zm8bc\"" Jul 21 14:31:10.023247 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.023232 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fv4cl\"/\"kube-root-ca.crt\"" Jul 21 14:31:10.023333 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.023282 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fv4cl\"/\"openshift-service-ca.crt\"" Jul 21 14:31:10.028947 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.028927 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p"] Jul 21 14:31:10.148243 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.148205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-proc\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.148438 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.148266 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rvw\" (UniqueName: \"kubernetes.io/projected/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-kube-api-access-n4rvw\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.148438 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.148364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-podres\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.148438 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.148414 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-lib-modules\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.148585 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.148504 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-sys\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249352 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-proc\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249639 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rvw\" (UniqueName: \"kubernetes.io/projected/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-kube-api-access-n4rvw\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249639 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-podres\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249639 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249459 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-proc\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249639 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-lib-modules\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249639 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-sys\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249639 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249611 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-lib-modules\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249871 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249638 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-sys\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.249871 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.249646 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-podres\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.259427 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.259402 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rvw\" (UniqueName: \"kubernetes.io/projected/4bd7d62c-19fb-4ac1-a369-a665cec1d8c0-kube-api-access-n4rvw\") pod \"perf-node-gather-daemonset-vq69p\" (UID: \"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0\") " pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.330712 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.330616 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.462828 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.462803 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p"] Jul 21 14:31:10.465150 ip-10-0-129-45 kubenswrapper[2569]: W0721 14:31:10.465121 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4bd7d62c_19fb_4ac1_a369_a665cec1d8c0.slice/crio-83f8dcedb003acafb38345ef45d2dceea18ab1ba301336844ba2250df5dd6cba WatchSource:0}: Error finding container 83f8dcedb003acafb38345ef45d2dceea18ab1ba301336844ba2250df5dd6cba: Status 404 returned error can't find the container with id 83f8dcedb003acafb38345ef45d2dceea18ab1ba301336844ba2250df5dd6cba Jul 21 14:31:10.688117 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.688025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" event={"ID":"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0","Type":"ContainerStarted","Data":"814dea0999e6d8af9211e285f0bb21a211d4a5c6536a7702ad0f0cc97e5f90d1"} Jul 21 14:31:10.688117 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.688090 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:10.688117 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.688115 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" event={"ID":"4bd7d62c-19fb-4ac1-a369-a665cec1d8c0","Type":"ContainerStarted","Data":"83f8dcedb003acafb38345ef45d2dceea18ab1ba301336844ba2250df5dd6cba"} Jul 21 14:31:10.705745 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:10.705698 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" podStartSLOduration=1.7056837 podStartE2EDuration="1.7056837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:31:10.703213495 +0000 UTC m=+2803.741283417" watchObservedRunningTime="2026-07-21 14:31:10.7056837 +0000 UTC m=+2803.743753621" Jul 21 14:31:11.623818 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:11.623759 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-czwsf_2f0ef78a-f8ca-4c56-88b5-4c8e54e09fde/download-server/0.log" Jul 21 14:31:13.522701 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:13.522664 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w6phj_cd056307-610f-48de-9f6a-7c0028921262/dns/0.log" Jul 21 14:31:13.545806 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:13.545772 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w6phj_cd056307-610f-48de-9f6a-7c0028921262/kube-rbac-proxy/0.log" Jul 21 14:31:13.575726 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:13.575703 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5gg2v_30c50200-8929-4ef8-b57c-79638beebc58/dns-node-resolver/0.log" Jul 21 14:31:14.300621 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:14.300585 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tfwjv_6c088994-9870-4c82-9f4e-25e99e2c348d/node-ca/0.log" Jul 21 14:31:15.454027 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:15.453986 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55c6b9f88d-9hq54_4119d337-191a-4129-8626-ed2505a0854e/kube-auth-proxy/0.log" Jul 21 14:31:15.582067 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:15.582036 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-prk4n_e9761259-3cef-4858-824d-ba2157670e1f/payload-pre-processing/0.log" Jul 21 14:31:16.378723 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:16.378691 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhvwx_f11f3626-cec0-42a7-a05d-3bde8f5e4888/serve-healthcheck-canary/0.log" Jul 21 14:31:16.703804 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:16.703734 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fv4cl/perf-node-gather-daemonset-vq69p" Jul 21 14:31:16.952194 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:16.952164 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-82f8j_6fcd41c3-839c-44d9-8085-84ce0f5c5086/kube-rbac-proxy/0.log" Jul 21 14:31:16.975606 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:16.975492 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-82f8j_6fcd41c3-839c-44d9-8085-84ce0f5c5086/exporter/0.log" Jul 21 14:31:16.999276 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:16.999242 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-82f8j_6fcd41c3-839c-44d9-8085-84ce0f5c5086/extractor/0.log" Jul 21 14:31:20.068314 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:20.068284 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b6bd44f67-rh5qr_64f132c4-bccf-47f2-9c9e-4205e33f5a83/manager/0.log" Jul 21 14:31:20.247199 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:20.247164 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d98d8798d-ccn85_fb6bd71d-faad-4dda-8283-b2887028bbe5/manager/0.log" Jul 21 14:31:20.270919 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:20.270892 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d98d8798d-ccn85_fb6bd71d-faad-4dda-8283-b2887028bbe5/copy-manifests/0.log" Jul 21 14:31:21.804335 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:21.804221 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55c6b9f88d-9hq54_4119d337-191a-4129-8626-ed2505a0854e/kube-auth-proxy/0.log" Jul 21 14:31:21.844489 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:21.844460 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-686f4b4fc5-vcn4z_818687d7-7030-4ae3-bc89-36a5d6713319/manager/0.log" Jul 21 14:31:21.902670 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:21.902637 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-prk4n_e9761259-3cef-4858-824d-ba2157670e1f/payload-pre-processing/0.log" Jul 21 14:31:23.326752 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:23.326727 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55c6b9f88d-9hq54_4119d337-191a-4129-8626-ed2505a0854e/kube-auth-proxy/0.log" Jul 21 14:31:23.449974 ip-10-0-129-45 kubenswrapper[2569]: I0721 14:31:23.449932 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-prk4n_e9761259-3cef-4858-824d-ba2157670e1f/payload-pre-processing/0.log"