Jul 06 22:27:44.268471 ip-10-0-136-56 systemd[1]: Starting Kubernetes Kubelet... Jul 06 22:27:44.708587 ip-10-0-136-56 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:27:44.708587 ip-10-0-136-56 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 22:27:44.708587 ip-10-0-136-56 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:27:44.708587 ip-10-0-136-56 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 22:27:44.708587 ip-10-0-136-56 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:27:44.710911 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.710822 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 22:27:44.716928 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716903 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:27:44.716928 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716923 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:27:44.716928 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716932 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:27:44.716928 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716935 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716940 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716943 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716948 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716951 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716954 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716957 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716960 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716963 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716967 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716969 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716975 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716978 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716981 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716984 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716987 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716990 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716993 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.716997 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717000 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:27:44.717100 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717003 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717006 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717008 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717011 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717017 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717020 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717023 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717026 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717029 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717032 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717034 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717037 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717040 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717043 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717047 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717050 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717454 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717458 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717461 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:27:44.717669 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717463 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717470 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717474 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717478 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717481 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717485 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717488 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717491 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717495 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717500 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717503 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717507 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717509 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717512 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717517 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717521 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717524 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717527 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:27:44.718209 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717530 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717535 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717541 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717544 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717547 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717549 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717553 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717555 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717558 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717563 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717567 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717571 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717576 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717579 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717583 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717588 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717592 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717594 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717597 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:27:44.718709 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717600 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717603 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717606 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717609 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717612 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717615 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717618 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717621 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717626 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717629 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717632 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717635 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717638 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717640 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717643 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717646 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717649 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717652 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717655 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.717657 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:27:44.719194 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718902 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718910 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718914 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718916 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718919 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718926 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718929 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718933 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718936 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718939 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718942 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718944 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718947 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718950 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718954 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718957 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718960 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718966 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718969 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718972 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:27:44.719742 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718975 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718978 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718981 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718984 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718987 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718990 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718992 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718995 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.718998 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719003 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719007 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719010 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719013 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719016 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719021 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719025 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719028 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719032 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719035 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:27:44.720273 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719039 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719044 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719051 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719054 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719057 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719061 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719064 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719067 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719070 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719073 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719076 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719079 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719082 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719085 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719089 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719095 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719098 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719101 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719104 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:27:44.720802 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719106 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719109 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719112 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719115 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719119 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719122 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719125 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719128 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719133 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719136 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719139 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719142 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719145 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719147 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719151 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719154 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719157 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719160 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719164 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719167 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:27:44.721318 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719173 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719176 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719179 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719182 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719185 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719188 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719191 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719194 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719201 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719204 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719207 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719210 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719215 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719219 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719222 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719224 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719227 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719230 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719233 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719252 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:27:44.721836 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719255 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719917 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719923 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719927 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719933 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719936 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719939 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719942 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719945 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719948 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719951 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719955 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719958 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719961 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719964 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719967 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719973 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719976 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719979 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719982 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:27:44.722367 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719985 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719988 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719993 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.719997 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720000 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720003 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720007 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720010 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720012 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720019 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720023 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720026 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720029 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720032 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720035 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720038 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720041 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720044 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720046 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720054 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:27:44.722904 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720056 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720062 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720065 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720068 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720070 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720073 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720076 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720079 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720083 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720087 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720090 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720092 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720096 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720101 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720104 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720107 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720110 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720113 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720116 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:27:44.723442 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720119 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720122 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720125 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720127 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720130 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720133 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720136 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720141 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720144 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720147 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720150 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720152 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720158 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720162 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720165 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720167 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720170 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720174 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720177 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720180 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:27:44.723955 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720185 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720188 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720191 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720195 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720198 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720201 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720203 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720206 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720209 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720212 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720215 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720217 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720223 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720226 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720230 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720233 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720253 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720256 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720259 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:27:44.724509 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720262 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.720265 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720497 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720512 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720522 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720653 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720675 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720681 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720688 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720695 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720707 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720713 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720719 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720724 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720730 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720735 2580 flags.go:64] FLAG: --cgroup-root="" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720740 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720745 2580 flags.go:64] FLAG: --client-ca-file="" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720750 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720755 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720762 2580 flags.go:64] FLAG: --cluster-domain="" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720767 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720772 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720777 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 22:27:44.725002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720783 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720789 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720794 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720799 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720805 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720810 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720815 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720820 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720825 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720830 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720848 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720854 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720860 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720867 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720875 2580 flags.go:64] FLAG: --enable-server="true" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720880 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720891 2580 flags.go:64] FLAG: --event-burst="100" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720899 2580 flags.go:64] FLAG: --event-qps="50" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720906 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720913 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720919 2580 flags.go:64] FLAG: --eviction-hard="" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720925 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720929 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720932 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720936 2580 flags.go:64] FLAG: --eviction-soft="" Jul 06 22:27:44.725649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720939 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720941 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720945 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720947 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720951 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720954 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720957 2580 flags.go:64] FLAG: --feature-gates="" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720960 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720964 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720967 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720970 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720973 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720976 2580 flags.go:64] FLAG: --help="false" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720980 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-136-56.ec2.internal" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720983 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720986 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720989 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.720994 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721000 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721003 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721006 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721010 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721013 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 22:27:44.726344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721016 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721020 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721023 2580 flags.go:64] FLAG: --kube-reserved="" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721026 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721029 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721033 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721035 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721039 2580 flags.go:64] FLAG: --lock-file="" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721041 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721045 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721048 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721059 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721063 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721066 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721069 2580 flags.go:64] FLAG: --logging-format="text" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721072 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721075 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721078 2580 flags.go:64] FLAG: --manifest-url="" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721081 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721087 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721090 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721094 2580 flags.go:64] FLAG: --max-pods="110" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721097 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721100 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721103 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 22:27:44.726962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721106 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721110 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721114 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721118 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721126 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721129 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721132 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721135 2580 flags.go:64] FLAG: --pod-cidr="" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721138 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721144 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721147 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721150 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721153 2580 flags.go:64] FLAG: --port="10250" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721156 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721159 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b8ac31b7ff33db57" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721163 2580 flags.go:64] FLAG: --qos-reserved="" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721166 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721170 2580 flags.go:64] FLAG: --register-node="true" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721173 2580 flags.go:64] FLAG: --register-with-taints="" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721177 2580 flags.go:64] FLAG: --registry-burst="10" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721180 2580 flags.go:64] FLAG: --registry-qps="5" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721183 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721186 2580 flags.go:64] FLAG: --reserved-memory="" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721190 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721194 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 22:27:44.727622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721197 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721200 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721218 2580 flags.go:64] FLAG: --runonce="false" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721223 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721227 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721230 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721233 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721236 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721239 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721244 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721248 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721251 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721254 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721257 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721260 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721263 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721266 2580 flags.go:64] FLAG: --system-cgroups="" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721269 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721276 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721279 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721282 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721286 2580 flags.go:64] FLAG: --tls-min-version="" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721289 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721292 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721294 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 22:27:44.728281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721298 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721301 2580 flags.go:64] FLAG: --v="2" Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721306 2580 flags.go:64] FLAG: --version="false" Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721312 2580 flags.go:64] FLAG: --vmodule="" Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721316 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721319 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721442 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721447 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721451 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721455 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721459 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721462 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721466 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721469 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721472 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721475 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721481 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721484 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721486 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721489 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:27:44.728933 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721492 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721495 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721498 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721500 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721503 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721506 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721509 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721512 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721514 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721517 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721520 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721523 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721526 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721529 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721531 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721534 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721538 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721540 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721543 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721546 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:27:44.729474 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721548 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721551 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721554 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721556 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721559 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721562 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721565 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721568 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721573 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721576 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721578 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721581 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721584 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721586 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721589 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721592 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721594 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721597 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721600 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721603 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:27:44.730028 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721605 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721609 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721613 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721616 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721619 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721622 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721625 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721627 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721630 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721637 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721640 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721643 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721645 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721649 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721653 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721656 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721659 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721662 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721666 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:27:44.730567 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721668 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721673 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721676 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721680 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721683 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721686 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721688 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721691 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721694 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721697 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721700 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721702 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721705 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721708 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721711 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721713 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721716 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721719 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721722 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721724 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:27:44.731071 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721728 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721730 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721733 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721736 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721739 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.721741 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.721747 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.729390 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.729404 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.730314 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.730323 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730385 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730390 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730393 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730396 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:27:44.731601 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730400 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730403 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730407 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730410 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730413 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730416 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730418 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730422 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730424 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730427 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730430 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730433 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730435 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730438 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730441 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730443 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730448 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730451 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730455 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:27:44.732012 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730458 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730461 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730464 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730467 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730470 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730473 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730478 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730481 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730484 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730487 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730490 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730493 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730495 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730498 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730501 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730504 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730506 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730509 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730512 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730515 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:27:44.732600 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730517 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730520 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730523 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730525 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730528 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730531 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730534 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730537 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730540 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730543 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730546 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730549 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730552 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730555 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730558 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730561 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730564 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730567 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730570 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730574 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:27:44.733111 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730577 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730579 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730582 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730585 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730588 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730590 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730593 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730596 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730598 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730601 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730604 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730607 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730610 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730612 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730617 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730620 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730624 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730628 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730632 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:27:44.733634 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730635 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730639 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730642 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730644 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730647 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730650 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730653 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730656 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730659 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730662 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730665 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730668 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730671 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730674 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730677 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730680 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:27:44.734128 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730682 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.730688 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730809 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730814 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730818 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730821 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730825 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730827 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730831 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730834 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730837 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730839 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730842 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730844 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730847 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730850 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:27:44.734561 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730852 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730855 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730858 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730860 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730863 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730866 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730869 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730871 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730874 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730877 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730879 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730882 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730885 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730888 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730891 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730894 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730898 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730902 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730906 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:27:44.734999 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730909 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730912 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730915 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730918 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730921 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730924 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730927 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730930 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730933 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730935 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730938 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730941 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730943 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730946 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730949 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730951 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730954 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730957 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730959 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730962 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:27:44.735517 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730965 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730968 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730970 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730973 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730976 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730978 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730981 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730984 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730987 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730990 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730994 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.730998 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731001 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731003 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731006 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731012 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731014 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731017 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731020 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731022 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:27:44.736031 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731025 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731027 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731032 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731035 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731039 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731042 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731045 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731048 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731050 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731053 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731055 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731067 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731072 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731075 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731078 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731081 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731084 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731087 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731090 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:27:44.736643 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731093 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731096 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731100 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731102 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731105 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731108 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:44.731111 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.731116 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.731251 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.733326 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.734988 2580 server.go:1019] "Starting client certificate rotation" Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.735093 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:27:44.737133 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.735131 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:27:44.738619 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.738603 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:27:44.740669 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.740650 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:27:44.750668 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.750650 2580 log.go:25] "Validated CRI v1 runtime API" Jul 06 22:27:44.753697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.753684 2580 log.go:25] "Validated CRI v1 image API" Jul 06 22:27:44.754636 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.754617 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 22:27:44.758486 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.758459 2580 fs.go:135] Filesystem UUIDs: map[03036dc1-85fd-4d76-a6a2-129e72b2adee:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c667230c-c714-401e-b24a-cbc52d54bb55:/dev/nvme0n1p4] Jul 06 22:27:44.758575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.758484 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 22:27:44.765121 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.765006 2580 manager.go:216] Machine: {Timestamp:2026-07-06 22:27:44.763613497 +0000 UTC m=+0.373614699 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101213 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b7de95fbc093f0b20dd425090b94 SystemUUID:ec25b7de-95fb-c093-f0b2-0dd425090b94 BootID:e5e81223-3d6b-42d3-b15d-97c9cf832671 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4e:4b:2a:4a:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4e:4b:2a:4a:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:19:33:ac:ef:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 22:27:44.765121 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.765115 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 22:27:44.765264 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.765198 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 22:27:44.766730 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.766702 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 22:27:44.766876 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.766733 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-56.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 22:27:44.767640 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.767628 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 22:27:44.767688 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.767645 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 22:27:44.767688 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.767653 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:27:44.768552 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.768541 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:27:44.768940 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.768918 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:27:44.769761 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.769749 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 22:27:44.772911 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.772899 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:27:44.773034 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.773023 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 22:27:44.773074 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.773056 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 06 22:27:44.773074 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.773068 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 22:27:44.773130 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.773093 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 22:27:44.773130 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.773107 2580 kubelet.go:398] "Adding apiserver pod source" Jul 06 22:27:44.773130 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.773120 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 22:27:44.773507 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.773495 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:27:44.773553 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.773513 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:27:44.775958 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.775932 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 22:27:44.778163 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.778150 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 22:27:44.778217 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.778182 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 22:27:44.779498 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779483 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 22:27:44.779498 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779502 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779508 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779515 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779524 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779530 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779539 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779544 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779552 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779559 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779568 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 22:27:44.779590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.779574 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 22:27:44.780204 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.780194 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 22:27:44.780204 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.780204 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 22:27:44.783783 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.783764 2580 server.go:1268] "Started kubelet" Jul 06 22:27:44.783883 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.783815 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 22:27:44.784523 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.783913 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 22:27:44.784660 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.784555 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 22:27:44.784998 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.784981 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-56.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 22:27:44.785156 ip-10-0-136-56 systemd[1]: Started Kubernetes Kubelet. Jul 06 22:27:44.785322 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.785225 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 22:27:44.785322 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.785242 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 22:27:44.786308 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.786288 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 22:27:44.791536 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.791506 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 22:27:44.791644 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.791581 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 06 22:27:44.795624 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.795604 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 22:27:44.795624 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.795616 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 22:27:44.795770 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.795710 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:44.796365 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796247 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 22:27:44.796365 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796250 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 22:27:44.796365 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796308 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 22:27:44.796614 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796400 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 06 22:27:44.796614 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796410 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 06 22:27:44.796719 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796653 2580 factory.go:55] Registering systemd factory Jul 06 22:27:44.796719 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796669 2580 factory.go:223] Registration of the systemd container factory successfully Jul 06 22:27:44.796956 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796929 2580 factory.go:153] Registering CRI-O factory Jul 06 22:27:44.796956 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.796949 2580 factory.go:223] Registration of the crio container factory successfully Jul 06 22:27:44.797068 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.797011 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 22:27:44.797068 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.797041 2580 factory.go:103] Registering Raw factory Jul 06 22:27:44.797068 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.797056 2580 manager.go:1196] Started watching for new ooms in manager Jul 06 22:27:44.797441 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.797427 2580 manager.go:349] Starting recovery of all containers Jul 06 22:27:44.798134 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.796854 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-56.ec2.internal.18bfd3cf4e158ed0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-56.ec2.internal,UID:ip-10-0-136-56.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-56.ec2.internal,},FirstTimestamp:2026-07-06 22:27:44.783732432 +0000 UTC m=+0.393733634,LastTimestamp:2026-07-06 22:27:44.783732432 +0000 UTC m=+0.393733634,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-56.ec2.internal,}" Jul 06 22:27:44.800952 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.800923 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 22:27:44.801048 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.801023 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 22:27:44.806718 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.806694 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9bq7" Jul 06 22:27:44.807197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.807183 2580 manager.go:354] Recovery completed Jul 06 22:27:44.814494 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.814470 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:27:44.816466 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.816419 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:27:44.816564 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.816484 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:27:44.816564 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.816500 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:27:44.816658 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.816636 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9bq7" Jul 06 22:27:44.817153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.817136 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 22:27:44.817153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.817153 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 22:27:44.817261 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.817170 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:27:44.817261 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.817203 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:27:44.817953 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.817929 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:27:44.818039 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.817972 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:27:44.818039 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.817988 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:27:44.818981 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.818908 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-56.ec2.internal.18bfd3cf50090af9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-56.ec2.internal,UID:ip-10-0-136-56.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-56.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-56.ec2.internal,},FirstTimestamp:2026-07-06 22:27:44.816466681 +0000 UTC m=+0.426467888,LastTimestamp:2026-07-06 22:27:44.816466681 +0000 UTC m=+0.426467888,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-56.ec2.internal,}" Jul 06 22:27:44.819410 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.819392 2580 policy_none.go:49] "None policy: Start" Jul 06 22:27:44.819490 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.819419 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 22:27:44.819490 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.819436 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 22:27:44.820694 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.820683 2580 policy_none.go:47] "Start" Jul 06 22:27:44.858717 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.858703 2580 manager.go:337] "Starting Device Plugin manager" Jul 06 22:27:44.858788 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.858738 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 22:27:44.858788 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.858753 2580 server.go:87] "Starting device plugin registration server" Jul 06 22:27:44.859056 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.859040 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 22:27:44.859128 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.859057 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 22:27:44.859128 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.859124 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 22:27:44.859304 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.859227 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 22:27:44.859304 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.859236 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 22:27:44.859809 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.859776 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 22:27:44.859883 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.859838 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:44.959715 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.959693 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:27:44.960948 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.960931 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:27:44.961025 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.960963 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:27:44.961025 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.960976 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:27:44.961025 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.961008 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-56.ec2.internal" Jul 06 22:27:44.969921 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.969903 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-56.ec2.internal" Jul 06 22:27:44.969987 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.969928 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-56.ec2.internal\": node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:44.990245 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.990224 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:44.995773 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.995749 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 22:27:44.997037 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.997017 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 22:27:44.997112 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.997058 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 22:27:44.997112 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:44.997081 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 22:27:44.997191 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:44.997121 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 22:27:45.001146 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.001130 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:27:45.091128 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.091102 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.097870 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.097848 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal"] Jul 06 22:27:45.097932 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.097923 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:27:45.098956 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.098939 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:27:45.099055 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.098984 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:27:45.099055 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.099002 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:27:45.100461 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.100444 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:27:45.100576 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.100559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.100636 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.100596 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:27:45.101308 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.101290 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:27:45.101361 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.101329 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:27:45.101361 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.101341 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:27:45.101427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.101294 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:27:45.101427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.101402 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:27:45.101427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.101413 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:27:45.102736 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.102720 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.102790 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.102747 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:27:45.103464 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.103448 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:27:45.103556 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.103475 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:27:45.103556 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.103489 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:27:45.128023 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.128003 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-56.ec2.internal\" not found" node="ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.132785 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.132767 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-56.ec2.internal\" not found" node="ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.192133 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.192105 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.198925 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.198902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ac28a16a2b80991296137c48fab92c8-config\") pod \"kube-apiserver-proxy-ip-10-0-136-56.ec2.internal\" (UID: \"0ac28a16a2b80991296137c48fab92c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.199008 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.198932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fbe9855291772162bbc6c68a6eba857-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal\" (UID: \"8fbe9855291772162bbc6c68a6eba857\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.199008 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.198951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fbe9855291772162bbc6c68a6eba857-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal\" (UID: \"8fbe9855291772162bbc6c68a6eba857\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.293075 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.292993 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.299874 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.299853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fbe9855291772162bbc6c68a6eba857-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal\" (UID: \"8fbe9855291772162bbc6c68a6eba857\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.299951 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.299881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fbe9855291772162bbc6c68a6eba857-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal\" (UID: \"8fbe9855291772162bbc6c68a6eba857\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.299951 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.299902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ac28a16a2b80991296137c48fab92c8-config\") pod \"kube-apiserver-proxy-ip-10-0-136-56.ec2.internal\" (UID: \"0ac28a16a2b80991296137c48fab92c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.300101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.299955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ac28a16a2b80991296137c48fab92c8-config\") pod \"kube-apiserver-proxy-ip-10-0-136-56.ec2.internal\" (UID: \"0ac28a16a2b80991296137c48fab92c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.300101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.299983 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fbe9855291772162bbc6c68a6eba857-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal\" (UID: \"8fbe9855291772162bbc6c68a6eba857\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.300101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.299982 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fbe9855291772162bbc6c68a6eba857-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal\" (UID: \"8fbe9855291772162bbc6c68a6eba857\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.400791 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.400746 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.432197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.432140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.436447 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.436262 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" Jul 06 22:27:45.501695 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.501661 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.602116 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.602009 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.671544 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.671514 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:27:45.702210 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.702174 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.734203 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.734153 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 22:27:45.734772 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.734333 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:27:45.734772 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.734365 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:27:45.802807 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.802759 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.819736 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.819694 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 22:22:45 +0000 UTC" deadline="2028-03-06 11:59:53.893429915 +0000 UTC" Jul 06 22:27:45.819736 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.819728 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14605h32m8.07370586s" Jul 06 22:27:45.903483 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:45.903452 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:45.973690 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:45.973657 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac28a16a2b80991296137c48fab92c8.slice/crio-673ce903d470b30e85c9a1fa5433d2742a354a4a4589fc226f4942742a7f2da6 WatchSource:0}: Error finding container 673ce903d470b30e85c9a1fa5433d2742a354a4a4589fc226f4942742a7f2da6: Status 404 returned error can't find the container with id 673ce903d470b30e85c9a1fa5433d2742a354a4a4589fc226f4942742a7f2da6 Jul 06 22:27:45.974018 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:45.973999 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbe9855291772162bbc6c68a6eba857.slice/crio-f2138faff6122fc066d65944edbc2248ed3c26cb468154809e94575ae61671c6 WatchSource:0}: Error finding container f2138faff6122fc066d65944edbc2248ed3c26cb468154809e94575ae61671c6: Status 404 returned error can't find the container with id f2138faff6122fc066d65944edbc2248ed3c26cb468154809e94575ae61671c6 Jul 06 22:27:45.982158 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.982137 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:27:45.983567 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:45.983546 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:27:46.000678 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.000625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" event={"ID":"8fbe9855291772162bbc6c68a6eba857","Type":"ContainerStarted","Data":"f2138faff6122fc066d65944edbc2248ed3c26cb468154809e94575ae61671c6"} Jul 06 22:27:46.001815 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.001787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" event={"ID":"0ac28a16a2b80991296137c48fab92c8","Type":"ContainerStarted","Data":"673ce903d470b30e85c9a1fa5433d2742a354a4a4589fc226f4942742a7f2da6"} Jul 06 22:27:46.004093 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.004071 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:46.104242 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.104210 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-56.ec2.internal\" not found" Jul 06 22:27:46.108455 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.108437 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:27:46.196804 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.196770 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" Jul 06 22:27:46.210318 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.210293 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:27:46.211450 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.211436 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" Jul 06 22:27:46.221261 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.221243 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:27:46.774671 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.774635 2580 apiserver.go:52] "Watching apiserver" Jul 06 22:27:46.784430 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.784249 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 22:27:46.784753 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.784728 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9","openshift-cluster-node-tuning-operator/tuned-5qqwd","openshift-dns/node-resolver-hjcm5","openshift-image-registry/node-ca-t6mx4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal","openshift-multus/multus-additional-cni-plugins-pwl2c","openshift-multus/multus-qmvqb","kube-system/global-pull-secret-syncer-4qqdf","kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal","openshift-multus/network-metrics-daemon-hcq92","openshift-network-diagnostics/network-check-target-8wq78","openshift-network-operator/iptables-alerter-h2ss2","openshift-ovn-kubernetes/ovnkube-node-vqkmg","kube-system/konnectivity-agent-9fw9r"] Jul 06 22:27:46.786432 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.786406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.787953 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.787931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:46.788057 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.788023 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:27:46.789171 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.789150 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.789600 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.789579 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 22:27:46.789785 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.789765 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 22:27:46.789856 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.789778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 22:27:46.789912 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.789863 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zfhkr\"" Jul 06 22:27:46.789912 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.789776 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 22:27:46.790422 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.790402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.792862 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.792838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:27:46.793023 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.792997 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 22:27:46.793410 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.793362 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 22:27:46.793502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.793447 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hxkgs\"" Jul 06 22:27:46.793574 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.793566 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 22:27:46.793820 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.793667 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 22:27:46.793820 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.793699 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 22:27:46.794395 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.794373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fw2k7\"" Jul 06 22:27:46.795036 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.795020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.795721 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.795700 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-94468\"" Jul 06 22:27:46.795822 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.795770 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 22:27:46.795923 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.795904 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 22:27:46.796539 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.796474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.796609 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.796572 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:27:46.796981 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.796885 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 22:27:46.797755 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.797734 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 22:27:46.798050 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.798034 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 22:27:46.798347 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.798330 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r8m4k\"" Jul 06 22:27:46.798508 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.798481 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.800138 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.800121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.801965 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.801942 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wbfb8\"" Jul 06 22:27:46.802172 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.802156 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 22:27:46.803831 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.803711 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 22:27:46.804191 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.804173 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:27:46.804641 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.804623 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:46.804754 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.804727 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:27:46.804900 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.804877 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 22:27:46.805364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.805342 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 22:27:46.805659 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.805641 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wrmwx\"" Jul 06 22:27:46.805912 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.805895 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 22:27:46.806694 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.806263 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 22:27:46.806694 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.806568 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 22:27:46.808766 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.808742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:46.809022 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-cnibin\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.809120 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a9a3de8-111a-4398-925e-5d7faa235047-cni-binary-copy\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.809178 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-netns\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.809233 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-kubelet\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.809320 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wp8\" (UniqueName: \"kubernetes.io/projected/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-kube-api-access-c5wp8\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-k8s-cni-cncf-io\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0eca13d8-da9b-48b9-8ce8-f015af526993-hosts-file\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-serviceca\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-etc-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovnkube-config\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a9a3de8-111a-4398-925e-5d7faa235047-multus-daemon-config\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-os-release\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.809991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx6gj\" (UniqueName: \"kubernetes.io/projected/37b1c52d-069c-451a-a666-ea8837bf9f7a-kube-api-access-rx6gj\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.810021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:46.810305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.810056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-system-cni-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.810949 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.810309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eca13d8-da9b-48b9-8ce8-f015af526993-tmp-dir\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.811078 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-run-netns\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.811145 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-var-lib-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.811695 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-cnibin\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.811783 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.811783 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qpgb\" (UniqueName: \"kubernetes.io/projected/b0dc89e4-c797-4a01-bb1d-43b919feba1e-kube-api-access-6qpgb\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.811783 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-socket-dir-parent\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.811947 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f20455a2-a971-4b8f-bfab-879716005fbf-dbus\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.811947 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.811947 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:46.811947 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.811947 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-hostroot\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.811947 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36b3b092-dfcb-4578-87fb-fcfd9937d07d-konnectivity-ca\") pod \"konnectivity-agent-9fw9r\" (UID: \"36b3b092-dfcb-4578-87fb-fcfd9937d07d\") " pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.811970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-node-log\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-multus-certs\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-os-release\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-ovn\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knbqw\" (UniqueName: \"kubernetes.io/projected/2a9a3de8-111a-4398-925e-5d7faa235047-kube-api-access-knbqw\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-systemd\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-cni-netd\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-cni-bin\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.812268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-slash\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovn-node-metrics-cert\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-cni-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-conf-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-host\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-kubelet\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovnkube-script-lib\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-system-cni-dir\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-etc-kubernetes\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t44zt\" (UniqueName: \"kubernetes.io/projected/0eca13d8-da9b-48b9-8ce8-f015af526993-kube-api-access-t44zt\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36b3b092-dfcb-4578-87fb-fcfd9937d07d-agent-certs\") pod \"konnectivity-agent-9fw9r\" (UID: \"36b3b092-dfcb-4578-87fb-fcfd9937d07d\") " pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-systemd-units\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-cni-bin\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-cni-multus\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f20455a2-a971-4b8f-bfab-879716005fbf-kubelet-config\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-log-socket\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.812767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-env-overrides\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.813502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812727 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.813502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812783 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wg2n6\"" Jul 06 22:27:46.813502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812980 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 22:27:46.813502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.812996 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:27:46.813502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.813024 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:27:46.813502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.813146 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 22:27:46.813502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.813188 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 22:27:46.813502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.813359 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 22:27:46.814193 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.814175 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lws76\"" Jul 06 22:27:46.814300 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.814225 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 22:27:46.852081 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.852046 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cnbfj" Jul 06 22:27:46.860863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.860798 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cnbfj" Jul 06 22:27:46.897634 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.897609 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 22:27:46.913478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f20455a2-a971-4b8f-bfab-879716005fbf-dbus\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.913636 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.913636 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:46.913636 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e431e4b7-3383-4431-ad97-fecdfa7239b5-iptables-alerter-script\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:46.913808 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.913808 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-hostroot\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.913808 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.913684 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:46.913808 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36b3b092-dfcb-4578-87fb-fcfd9937d07d-konnectivity-ca\") pod \"konnectivity-agent-9fw9r\" (UID: \"36b3b092-dfcb-4578-87fb-fcfd9937d07d\") " pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:27:46.913808 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7c3375-87b1-434b-9b11-dc4f05520e92-tmp\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.913808 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.913759 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret podName:f20455a2-a971-4b8f-bfab-879716005fbf nodeName:}" failed. No retries permitted until 2026-07-06 22:27:47.413736164 +0000 UTC m=+3.023737388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret") pod "global-pull-secret-syncer-4qqdf" (UID: "f20455a2-a971-4b8f-bfab-879716005fbf") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913804 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-hostroot\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913814 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913708 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f20455a2-a971-4b8f-bfab-879716005fbf-dbus\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-node-log\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-multus-certs\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.913987 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-node-log\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-run\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-tuned\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914039 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-multus-certs\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914005 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcfh\" (UniqueName: \"kubernetes.io/projected/ce7c3375-87b1-434b-9b11-dc4f05520e92-kube-api-access-rdcfh\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-device-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-os-release\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-lib-modules\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.914153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4cqw\" (UniqueName: \"kubernetes.io/projected/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-kube-api-access-x4cqw\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-ovn\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914214 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-os-release\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knbqw\" (UniqueName: \"kubernetes.io/projected/2a9a3de8-111a-4398-925e-5d7faa235047-kube-api-access-knbqw\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-sys\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-systemd\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914377 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36b3b092-dfcb-4578-87fb-fcfd9937d07d-konnectivity-ca\") pod \"konnectivity-agent-9fw9r\" (UID: \"36b3b092-dfcb-4578-87fb-fcfd9937d07d\") " pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-cni-netd\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-cni-bin\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914451 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-ovn\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914472 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-cni-netd\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-slash\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914498 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-slash\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914511 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-run-systemd\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914564 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-cni-bin\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.914970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914563 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovn-node-metrics-cert\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914593 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-cni-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-conf-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-kubernetes\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914748 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-conf-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-host\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914772 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-cni-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914785 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-host\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-kubelet\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovnkube-script-lib\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914854 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-kubelet\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-system-cni-dir\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914894 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-system-cni-dir\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-etc-kubernetes\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914890 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 22:27:46.915886 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t44zt\" (UniqueName: \"kubernetes.io/projected/0eca13d8-da9b-48b9-8ce8-f015af526993-kube-api-access-t44zt\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.914962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36b3b092-dfcb-4578-87fb-fcfd9937d07d-agent-certs\") pod \"konnectivity-agent-9fw9r\" (UID: \"36b3b092-dfcb-4578-87fb-fcfd9937d07d\") " pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-systemd-units\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915016 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-etc-kubernetes\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-cni-bin\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-cni-multus\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6dw\" (UniqueName: \"kubernetes.io/projected/e431e4b7-3383-4431-ad97-fecdfa7239b5-kube-api-access-2b6dw\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f20455a2-a971-4b8f-bfab-879716005fbf-kubelet-config\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-log-socket\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-env-overrides\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-cnibin\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a9a3de8-111a-4398-925e-5d7faa235047-cni-binary-copy\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-netns\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-kubelet\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wp8\" (UniqueName: \"kubernetes.io/projected/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-kube-api-access-c5wp8\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-systemd\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.916863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-k8s-cni-cncf-io\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915617 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-cni-bin\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0eca13d8-da9b-48b9-8ce8-f015af526993-hosts-file\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-serviceca\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915703 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-cni-multus\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysctl-d\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915762 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f20455a2-a971-4b8f-bfab-879716005fbf-kubelet-config\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e431e4b7-3383-4431-ad97-fecdfa7239b5-host-slash\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-log-socket\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-etc-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovnkube-config\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915966 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-env-overrides\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a9a3de8-111a-4398-925e-5d7faa235047-multus-daemon-config\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-os-release\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.917720 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6gj\" (UniqueName: \"kubernetes.io/projected/37b1c52d-069c-451a-a666-ea8837bf9f7a-kube-api-access-rx6gj\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-system-cni-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-netns\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eca13d8-da9b-48b9-8ce8-f015af526993-tmp-dir\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916279 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysctl-conf\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.915536 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovnkube-script-lib\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916388 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-os-release\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916215 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-cnibin\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916598 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eca13d8-da9b-48b9-8ce8-f015af526993-tmp-dir\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916660 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a9a3de8-111a-4398-925e-5d7faa235047-multus-daemon-config\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916693 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-systemd-units\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916714 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-etc-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916764 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916897 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37b1c52d-069c-451a-a666-ea8837bf9f7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916961 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-system-cni-dir\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.916991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a9a3de8-111a-4398-925e-5d7faa235047-cni-binary-copy\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.918661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917007 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-run-k8s-cni-cncf-io\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917044 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-host-var-lib-kubelet\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917058 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0eca13d8-da9b-48b9-8ce8-f015af526993-hosts-file\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48sxq\" (UniqueName: \"kubernetes.io/projected/7f6b6507-63df-4cff-8744-a52377707ff8-kube-api-access-48sxq\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-run-netns\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-var-lib-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-cnibin\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysconfig\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917326 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovnkube-config\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917342 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-host-run-netns\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-host\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917436 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-cnibin\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917588 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1c52d-069c-451a-a666-ea8837bf9f7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qpgb\" (UniqueName: \"kubernetes.io/projected/b0dc89e4-c797-4a01-bb1d-43b919feba1e-kube-api-access-6qpgb\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-socket-dir-parent\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-modprobe-d\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.919617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917885 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a9a3de8-111a-4398-925e-5d7faa235047-multus-socket-dir-parent\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.920489 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-var-lib-kubelet\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:46.920489 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917982 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-serviceca\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.920489 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.917454 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0dc89e4-c797-4a01-bb1d-43b919feba1e-var-lib-openvswitch\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.920858 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.920839 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:27:46.920972 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.920863 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:27:46.920972 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.920879 2580 projected.go:196] Error preparing data for projected volume kube-api-access-57jp5 for pod openshift-network-diagnostics/network-check-target-8wq78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:46.920972 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:46.920945 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5 podName:c9a54d7a-954e-405c-a0c5-8fb9a3d37077 nodeName:}" failed. No retries permitted until 2026-07-06 22:27:47.420926452 +0000 UTC m=+3.030927641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-57jp5" (UniqueName: "kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5") pod "network-check-target-8wq78" (UID: "c9a54d7a-954e-405c-a0c5-8fb9a3d37077") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:46.921767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.921432 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0dc89e4-c797-4a01-bb1d-43b919feba1e-ovn-node-metrics-cert\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.923016 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.922707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36b3b092-dfcb-4578-87fb-fcfd9937d07d-agent-certs\") pod \"konnectivity-agent-9fw9r\" (UID: \"36b3b092-dfcb-4578-87fb-fcfd9937d07d\") " pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:27:46.924073 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.924049 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knbqw\" (UniqueName: \"kubernetes.io/projected/2a9a3de8-111a-4398-925e-5d7faa235047-kube-api-access-knbqw\") pod \"multus-qmvqb\" (UID: \"2a9a3de8-111a-4398-925e-5d7faa235047\") " pod="openshift-multus/multus-qmvqb" Jul 06 22:27:46.925778 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.925733 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t44zt\" (UniqueName: \"kubernetes.io/projected/0eca13d8-da9b-48b9-8ce8-f015af526993-kube-api-access-t44zt\") pod \"node-resolver-hjcm5\" (UID: \"0eca13d8-da9b-48b9-8ce8-f015af526993\") " pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:46.927777 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.927756 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qpgb\" (UniqueName: \"kubernetes.io/projected/b0dc89e4-c797-4a01-bb1d-43b919feba1e-kube-api-access-6qpgb\") pod \"ovnkube-node-vqkmg\" (UID: \"b0dc89e4-c797-4a01-bb1d-43b919feba1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:46.928318 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.928295 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wp8\" (UniqueName: \"kubernetes.io/projected/e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e-kube-api-access-c5wp8\") pod \"node-ca-t6mx4\" (UID: \"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e\") " pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:46.937147 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:46.937121 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6gj\" (UniqueName: \"kubernetes.io/projected/37b1c52d-069c-451a-a666-ea8837bf9f7a-kube-api-access-rx6gj\") pod \"multus-additional-cni-plugins-pwl2c\" (UID: \"37b1c52d-069c-451a-a666-ea8837bf9f7a\") " pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:47.005352 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.005318 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:27:47.019085 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysconfig\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019085 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-host\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019331 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-modprobe-d\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019331 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-var-lib-kubelet\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019331 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019192 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysconfig\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019331 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-host\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019331 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e431e4b7-3383-4431-ad97-fecdfa7239b5-iptables-alerter-script\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:47.019331 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7c3375-87b1-434b-9b11-dc4f05520e92-tmp\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019331 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019331 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-var-lib-kubelet\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019314 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-modprobe-d\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-run\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019357 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-run\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-tuned\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcfh\" (UniqueName: \"kubernetes.io/projected/ce7c3375-87b1-434b-9b11-dc4f05520e92-kube-api-access-rdcfh\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-device-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-lib-modules\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4cqw\" (UniqueName: \"kubernetes.io/projected/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-kube-api-access-x4cqw\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-sys\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019561 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-device-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-kubernetes\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.019651 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:47.019697 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.019712 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs podName:4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433 nodeName:}" failed. No retries permitted until 2026-07-06 22:27:47.519691724 +0000 UTC m=+3.129692918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs") pod "network-metrics-daemon-hcq92" (UID: "4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019754 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019807 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-kubernetes\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019859 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6dw\" (UniqueName: \"kubernetes.io/projected/e431e4b7-3383-4431-ad97-fecdfa7239b5-kube-api-access-2b6dw\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019928 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-sys\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019935 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e431e4b7-3383-4431-ad97-fecdfa7239b5-iptables-alerter-script\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.019987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-systemd\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysctl-d\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020075 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-lib-modules\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e431e4b7-3383-4431-ad97-fecdfa7239b5-host-slash\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysctl-conf\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48sxq\" (UniqueName: \"kubernetes.io/projected/7f6b6507-63df-4cff-8744-a52377707ff8-kube-api-access-48sxq\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.020583 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020341 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f6b6507-63df-4cff-8744-a52377707ff8-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.021477 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-systemd\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.021477 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020408 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e431e4b7-3383-4431-ad97-fecdfa7239b5-host-slash\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:47.021477 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020499 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysctl-d\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.021477 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.020561 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-sysctl-conf\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.022601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.022574 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7c3375-87b1-434b-9b11-dc4f05520e92-tmp\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.023572 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.023549 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce7c3375-87b1-434b-9b11-dc4f05520e92-etc-tuned\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.030632 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.030500 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4cqw\" (UniqueName: \"kubernetes.io/projected/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-kube-api-access-x4cqw\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:47.030632 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.030599 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48sxq\" (UniqueName: \"kubernetes.io/projected/7f6b6507-63df-4cff-8744-a52377707ff8-kube-api-access-48sxq\") pod \"aws-ebs-csi-driver-node-tzjd9\" (UID: \"7f6b6507-63df-4cff-8744-a52377707ff8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.030971 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.030912 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6dw\" (UniqueName: \"kubernetes.io/projected/e431e4b7-3383-4431-ad97-fecdfa7239b5-kube-api-access-2b6dw\") pod \"iptables-alerter-h2ss2\" (UID: \"e431e4b7-3383-4431-ad97-fecdfa7239b5\") " pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:47.031466 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.031443 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcfh\" (UniqueName: \"kubernetes.io/projected/ce7c3375-87b1-434b-9b11-dc4f05520e92-kube-api-access-rdcfh\") pod \"tuned-5qqwd\" (UID: \"ce7c3375-87b1-434b-9b11-dc4f05520e92\") " pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.100916 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.100878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qmvqb" Jul 06 22:27:47.113335 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.113299 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hjcm5" Jul 06 22:27:47.125485 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.125458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t6mx4" Jul 06 22:27:47.132377 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.132348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:27:47.142307 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.142283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" Jul 06 22:27:47.151556 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.151523 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:27:47.159600 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.159579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" Jul 06 22:27:47.173423 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.173395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" Jul 06 22:27:47.179273 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.179250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h2ss2" Jul 06 22:27:47.423576 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.423492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:47.423576 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.423527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:47.423765 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.423636 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:27:47.423765 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.423643 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:47.423765 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.423709 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret podName:f20455a2-a971-4b8f-bfab-879716005fbf nodeName:}" failed. No retries permitted until 2026-07-06 22:27:48.423694506 +0000 UTC m=+4.033695695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret") pod "global-pull-secret-syncer-4qqdf" (UID: "f20455a2-a971-4b8f-bfab-879716005fbf") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:47.423765 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.423650 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:27:47.423765 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.423746 2580 projected.go:196] Error preparing data for projected volume kube-api-access-57jp5 for pod openshift-network-diagnostics/network-check-target-8wq78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:47.423954 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.423790 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5 podName:c9a54d7a-954e-405c-a0c5-8fb9a3d37077 nodeName:}" failed. No retries permitted until 2026-07-06 22:27:48.423777631 +0000 UTC m=+4.033778820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-57jp5" (UniqueName: "kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5") pod "network-check-target-8wq78" (UID: "c9a54d7a-954e-405c-a0c5-8fb9a3d37077") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:47.524841 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.524801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:47.525061 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.524985 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:47.525129 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:47.525071 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs podName:4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433 nodeName:}" failed. No retries permitted until 2026-07-06 22:27:48.525051221 +0000 UTC m=+4.135052427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs") pod "network-metrics-daemon-hcq92" (UID: "4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:47.863427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.863350 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:22:47 +0000 UTC" deadline="2028-02-14 20:05:27.790252141 +0000 UTC" Jul 06 22:27:47.863816 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:47.863432 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14109h37m39.926832197s" Jul 06 22:27:48.114872 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.114832 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eca13d8_da9b_48b9_8ce8_f015af526993.slice/crio-dc3a79eef7e4c98891c6e912089b2ae22fe6e521e9daa9bf0961fc94455f6d66 WatchSource:0}: Error finding container dc3a79eef7e4c98891c6e912089b2ae22fe6e521e9daa9bf0961fc94455f6d66: Status 404 returned error can't find the container with id dc3a79eef7e4c98891c6e912089b2ae22fe6e521e9daa9bf0961fc94455f6d66 Jul 06 22:27:48.116179 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.116149 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9a3de8_111a_4398_925e_5d7faa235047.slice/crio-220fea915c8ecbe61a8ce75773887699d2b962ec55069ac4da4098b074e8321a WatchSource:0}: Error finding container 220fea915c8ecbe61a8ce75773887699d2b962ec55069ac4da4098b074e8321a: Status 404 returned error can't find the container with id 220fea915c8ecbe61a8ce75773887699d2b962ec55069ac4da4098b074e8321a Jul 06 22:27:48.118351 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.118292 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dc89e4_c797_4a01_bb1d_43b919feba1e.slice/crio-dd5de9a47be4bdd5087d5b1a2827d337e314e93c5bb5f134a1b9d4ab57f5b52e WatchSource:0}: Error finding container dd5de9a47be4bdd5087d5b1a2827d337e314e93c5bb5f134a1b9d4ab57f5b52e: Status 404 returned error can't find the container with id dd5de9a47be4bdd5087d5b1a2827d337e314e93c5bb5f134a1b9d4ab57f5b52e Jul 06 22:27:48.123564 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.123537 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6b6507_63df_4cff_8744_a52377707ff8.slice/crio-fee166f2f0db283459e9ad8978f78dc728e25e7a941f27b7e976ecdb2b9a4697 WatchSource:0}: Error finding container fee166f2f0db283459e9ad8978f78dc728e25e7a941f27b7e976ecdb2b9a4697: Status 404 returned error can't find the container with id fee166f2f0db283459e9ad8978f78dc728e25e7a941f27b7e976ecdb2b9a4697 Jul 06 22:27:48.124823 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.124798 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b1c52d_069c_451a_a666_ea8837bf9f7a.slice/crio-75b3e2589d175f9e9dc1fd034d2319ded8cc61f25a4d7f7c296a3b0aa1ed411e WatchSource:0}: Error finding container 75b3e2589d175f9e9dc1fd034d2319ded8cc61f25a4d7f7c296a3b0aa1ed411e: Status 404 returned error can't find the container with id 75b3e2589d175f9e9dc1fd034d2319ded8cc61f25a4d7f7c296a3b0aa1ed411e Jul 06 22:27:48.126131 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.126106 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode431e4b7_3383_4431_ad97_fecdfa7239b5.slice/crio-c0cd659685845f8de1864ca6713957162fc93f798c4802a043029901aa573961 WatchSource:0}: Error finding container c0cd659685845f8de1864ca6713957162fc93f798c4802a043029901aa573961: Status 404 returned error can't find the container with id c0cd659685845f8de1864ca6713957162fc93f798c4802a043029901aa573961 Jul 06 22:27:48.127060 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.127035 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27b56a0_fd8d_4ee6_ac4b_4ae12c13864e.slice/crio-76f07dcdb62008edc77b9dd2da335da9af5c21550883e7d054cafc24671273e5 WatchSource:0}: Error finding container 76f07dcdb62008edc77b9dd2da335da9af5c21550883e7d054cafc24671273e5: Status 404 returned error can't find the container with id 76f07dcdb62008edc77b9dd2da335da9af5c21550883e7d054cafc24671273e5 Jul 06 22:27:48.128990 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.128894 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7c3375_87b1_434b_9b11_dc4f05520e92.slice/crio-0d3ca44051c4651cd0449c59ecd331bf510fd862ca0028e0e9c63399906f4ff6 WatchSource:0}: Error finding container 0d3ca44051c4651cd0449c59ecd331bf510fd862ca0028e0e9c63399906f4ff6: Status 404 returned error can't find the container with id 0d3ca44051c4651cd0449c59ecd331bf510fd862ca0028e0e9c63399906f4ff6 Jul 06 22:27:48.131164 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:27:48.130792 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b3b092_dfcb_4578_87fb_fcfd9937d07d.slice/crio-10817100187b2a000bf8d7fb4dfb6a866825d93bf2ee916bdb38c51678e25af2 WatchSource:0}: Error finding container 10817100187b2a000bf8d7fb4dfb6a866825d93bf2ee916bdb38c51678e25af2: Status 404 returned error can't find the container with id 10817100187b2a000bf8d7fb4dfb6a866825d93bf2ee916bdb38c51678e25af2 Jul 06 22:27:48.254494 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:48.254277 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:27:48.431372 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:48.431339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:48.431507 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:48.431379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:48.431549 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:48.431505 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:48.431549 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:48.431532 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:27:48.431609 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:48.431549 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:27:48.431609 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:48.431564 2580 projected.go:196] Error preparing data for projected volume kube-api-access-57jp5 for pod openshift-network-diagnostics/network-check-target-8wq78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:48.431609 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:48.431577 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret podName:f20455a2-a971-4b8f-bfab-879716005fbf nodeName:}" failed. No retries permitted until 2026-07-06 22:27:50.431558444 +0000 UTC m=+6.041559651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret") pod "global-pull-secret-syncer-4qqdf" (UID: "f20455a2-a971-4b8f-bfab-879716005fbf") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:48.431609 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:48.431602 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5 podName:c9a54d7a-954e-405c-a0c5-8fb9a3d37077 nodeName:}" failed. No retries permitted until 2026-07-06 22:27:50.431591329 +0000 UTC m=+6.041592525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-57jp5" (UniqueName: "kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5") pod "network-check-target-8wq78" (UID: "c9a54d7a-954e-405c-a0c5-8fb9a3d37077") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:48.532155 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:48.532123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:48.532478 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:48.532318 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:48.532478 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:48.532412 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs podName:4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433 nodeName:}" failed. No retries permitted until 2026-07-06 22:27:50.532389864 +0000 UTC m=+6.142391056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs") pod "network-metrics-daemon-hcq92" (UID: "4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:48.863699 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:48.863605 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:22:47 +0000 UTC" deadline="2028-02-15 01:58:16.101035564 +0000 UTC" Jul 06 22:27:48.863699 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:48.863666 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14115h30m27.237373271s" Jul 06 22:27:49.001009 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.000371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:49.001009 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.000385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:49.001009 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:49.000505 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:27:49.001009 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:49.000962 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:27:49.001706 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.001508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:49.001706 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:49.001609 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:27:49.017078 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.017004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerStarted","Data":"75b3e2589d175f9e9dc1fd034d2319ded8cc61f25a4d7f7c296a3b0aa1ed411e"} Jul 06 22:27:49.021546 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.021517 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fbe9855291772162bbc6c68a6eba857" containerID="1a944b9371fb79d26c388d4581edcb10e5f7ad421db1c64a540840b3b2462e7b" exitCode=0 Jul 06 22:27:49.021677 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.021598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" event={"ID":"8fbe9855291772162bbc6c68a6eba857","Type":"ContainerDied","Data":"1a944b9371fb79d26c388d4581edcb10e5f7ad421db1c64a540840b3b2462e7b"} Jul 06 22:27:49.027316 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.027248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9fw9r" event={"ID":"36b3b092-dfcb-4578-87fb-fcfd9937d07d","Type":"ContainerStarted","Data":"10817100187b2a000bf8d7fb4dfb6a866825d93bf2ee916bdb38c51678e25af2"} Jul 06 22:27:49.040695 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.040437 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" event={"ID":"7f6b6507-63df-4cff-8744-a52377707ff8","Type":"ContainerStarted","Data":"fee166f2f0db283459e9ad8978f78dc728e25e7a941f27b7e976ecdb2b9a4697"} Jul 06 22:27:49.048381 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.048352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"dd5de9a47be4bdd5087d5b1a2827d337e314e93c5bb5f134a1b9d4ab57f5b52e"} Jul 06 22:27:49.056697 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.056645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qmvqb" event={"ID":"2a9a3de8-111a-4398-925e-5d7faa235047","Type":"ContainerStarted","Data":"220fea915c8ecbe61a8ce75773887699d2b962ec55069ac4da4098b074e8321a"} Jul 06 22:27:49.070502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.070405 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hjcm5" event={"ID":"0eca13d8-da9b-48b9-8ce8-f015af526993","Type":"ContainerStarted","Data":"dc3a79eef7e4c98891c6e912089b2ae22fe6e521e9daa9bf0961fc94455f6d66"} Jul 06 22:27:49.088371 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.088296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" event={"ID":"0ac28a16a2b80991296137c48fab92c8","Type":"ContainerStarted","Data":"2da7d6a89488eecb33cca2a425c0005ac804a810b90b0dbe80e3d4063e3f301b"} Jul 06 22:27:49.106366 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.105430 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-56.ec2.internal" podStartSLOduration=3.105408131 podStartE2EDuration="3.105408131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:49.105190523 +0000 UTC m=+4.715191736" watchObservedRunningTime="2026-07-06 22:27:49.105408131 +0000 UTC m=+4.715409344" Jul 06 22:27:49.107997 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.107967 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" event={"ID":"ce7c3375-87b1-434b-9b11-dc4f05520e92","Type":"ContainerStarted","Data":"0d3ca44051c4651cd0449c59ecd331bf510fd862ca0028e0e9c63399906f4ff6"} Jul 06 22:27:49.112987 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.112923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h2ss2" event={"ID":"e431e4b7-3383-4431-ad97-fecdfa7239b5","Type":"ContainerStarted","Data":"c0cd659685845f8de1864ca6713957162fc93f798c4802a043029901aa573961"} Jul 06 22:27:49.130682 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:49.130573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t6mx4" event={"ID":"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e","Type":"ContainerStarted","Data":"76f07dcdb62008edc77b9dd2da335da9af5c21550883e7d054cafc24671273e5"} Jul 06 22:27:50.157213 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:50.157103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" event={"ID":"8fbe9855291772162bbc6c68a6eba857","Type":"ContainerStarted","Data":"601254dfe3755cfd977990fd0257b3de505d2f08ac4ff79589d98414616c8ac3"} Jul 06 22:27:50.450342 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:50.449393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:50.450342 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:50.449445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:50.450342 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.449609 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:27:50.450342 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.449627 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:27:50.450342 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.449641 2580 projected.go:196] Error preparing data for projected volume kube-api-access-57jp5 for pod openshift-network-diagnostics/network-check-target-8wq78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:50.450342 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.449699 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5 podName:c9a54d7a-954e-405c-a0c5-8fb9a3d37077 nodeName:}" failed. No retries permitted until 2026-07-06 22:27:54.44968112 +0000 UTC m=+10.059682330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-57jp5" (UniqueName: "kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5") pod "network-check-target-8wq78" (UID: "c9a54d7a-954e-405c-a0c5-8fb9a3d37077") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:50.450342 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.450095 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:50.450342 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.450146 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret podName:f20455a2-a971-4b8f-bfab-879716005fbf nodeName:}" failed. No retries permitted until 2026-07-06 22:27:54.450130698 +0000 UTC m=+10.060131890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret") pod "global-pull-secret-syncer-4qqdf" (UID: "f20455a2-a971-4b8f-bfab-879716005fbf") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:50.551363 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:50.550718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:50.551363 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.550915 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:50.551363 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.550978 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs podName:4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433 nodeName:}" failed. No retries permitted until 2026-07-06 22:27:54.550960627 +0000 UTC m=+10.160961821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs") pod "network-metrics-daemon-hcq92" (UID: "4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:50.998282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:50.998253 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:50.998464 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:50.998264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:50.998464 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.998373 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:27:50.998464 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:50.998402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:50.998616 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.998501 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:27:50.998616 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:50.998590 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:27:52.997625 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:52.997551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:52.997625 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:52.997551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:52.998156 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:52.997786 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:27:52.998156 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:52.998135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:52.998156 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:52.998250 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:27:52.998156 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:52.998352 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:27:54.482639 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:54.482454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:54.482639 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:54.482491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:54.482639 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.482615 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:54.483217 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.482677 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret podName:f20455a2-a971-4b8f-bfab-879716005fbf nodeName:}" failed. No retries permitted until 2026-07-06 22:28:02.48266027 +0000 UTC m=+18.092661461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret") pod "global-pull-secret-syncer-4qqdf" (UID: "f20455a2-a971-4b8f-bfab-879716005fbf") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:27:54.483217 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.482620 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:27:54.483217 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.482790 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:27:54.483217 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.482811 2580 projected.go:196] Error preparing data for projected volume kube-api-access-57jp5 for pod openshift-network-diagnostics/network-check-target-8wq78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:54.483217 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.482868 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5 podName:c9a54d7a-954e-405c-a0c5-8fb9a3d37077 nodeName:}" failed. No retries permitted until 2026-07-06 22:28:02.482850956 +0000 UTC m=+18.092852145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-57jp5" (UniqueName: "kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5") pod "network-check-target-8wq78" (UID: "c9a54d7a-954e-405c-a0c5-8fb9a3d37077") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:27:54.583264 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:54.583133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:54.583440 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.583297 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:54.583440 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.583366 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs podName:4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433 nodeName:}" failed. No retries permitted until 2026-07-06 22:28:02.583347875 +0000 UTC m=+18.193349063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs") pod "network-metrics-daemon-hcq92" (UID: "4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:27:54.999353 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:54.998569 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:54.999353 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.998693 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:27:54.999353 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:54.999048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:54.999353 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.999196 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:27:54.999353 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:54.999246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:54.999353 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:54.999314 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:27:56.998034 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:56.997590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:56.998034 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:56.997606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:56.998034 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:56.997741 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:27:56.998034 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:56.997861 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:27:56.998034 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:56.997926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:56.998034 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:56.997998 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:27:58.998632 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:58.998390 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:27:58.999326 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:58.998390 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:27:58.999326 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:58.998729 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:27:58.999326 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:58.998455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:27:58.999326 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:58.998811 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:27:58.999326 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:27:58.998872 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:27:59.177597 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.177530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerStarted","Data":"2fd7450b7489b255d3d4e8aba19b2d9c965c499fc554e331040a56e87e00805d"} Jul 06 22:27:59.179440 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.179410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9fw9r" event={"ID":"36b3b092-dfcb-4578-87fb-fcfd9937d07d","Type":"ContainerStarted","Data":"31fe01cf7641d8b80dd1a4679a76e5ecf6c3f175a8c1009f2bd869d12d8708a1"} Jul 06 22:27:59.181355 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.181273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" event={"ID":"7f6b6507-63df-4cff-8744-a52377707ff8","Type":"ContainerStarted","Data":"a56069a43e6c286e91bbbedc2f78a4a35abaeedc1b4271784dff8427f5ccf617"} Jul 06 22:27:59.183157 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.183131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hjcm5" event={"ID":"0eca13d8-da9b-48b9-8ce8-f015af526993","Type":"ContainerStarted","Data":"3d1a185142246d52a51e2c0faeefe3ffe5ea64020474a66ba17d8c5ff9ecb6fc"} Jul 06 22:27:59.184969 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.184944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" event={"ID":"ce7c3375-87b1-434b-9b11-dc4f05520e92","Type":"ContainerStarted","Data":"16ec10bb109f60ef7824961a8128d94166575a96cda05344892cde2027f29928"} Jul 06 22:27:59.186495 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.186470 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t6mx4" event={"ID":"e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e","Type":"ContainerStarted","Data":"0bc7e9ca2f6f8cd30df6457741f815e755ddd08c4e0851b134cf68319bd23d4a"} Jul 06 22:27:59.203141 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.203089 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-56.ec2.internal" podStartSLOduration=13.203072685 podStartE2EDuration="13.203072685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:50.173472891 +0000 UTC m=+5.783474104" watchObservedRunningTime="2026-07-06 22:27:59.203072685 +0000 UTC m=+14.813073895" Jul 06 22:27:59.231237 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.231185 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9fw9r" podStartSLOduration=4.096849931 podStartE2EDuration="14.231146716s" totalImagesPullingTime="10.134296785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:59.216182549 +0000 UTC m=+14.826183760" watchObservedRunningTime="2026-07-06 22:27:59.231146716 +0000 UTC m=+14.841147925" Jul 06 22:27:59.231536 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.231506 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t6mx4" podStartSLOduration=4.058617811 podStartE2EDuration="14.231499151s" totalImagesPullingTime="10.17288134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:59.230710255 +0000 UTC m=+14.840711465" watchObservedRunningTime="2026-07-06 22:27:59.231499151 +0000 UTC m=+14.841500361" Jul 06 22:27:59.248547 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:27:59.248425 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5qqwd" podStartSLOduration=4.049168723 podStartE2EDuration="14.248410628s" totalImagesPullingTime="10.199241905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:59.247628516 +0000 UTC m=+14.857629726" watchObservedRunningTime="2026-07-06 22:27:59.248410628 +0000 UTC m=+14.858411836" Jul 06 22:28:00.190349 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:00.190303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h2ss2" event={"ID":"e431e4b7-3383-4431-ad97-fecdfa7239b5","Type":"ContainerStarted","Data":"f22128614fda9d4a01ece2baa0bdd8a2e34552c104a64146c3dd2cb33b028a72"} Jul 06 22:28:00.205934 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:00.205887 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h2ss2" podStartSLOduration=5.055633777 podStartE2EDuration="15.205866798s" totalImagesPullingTime="10.150233021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:00.205812095 +0000 UTC m=+15.815813306" watchObservedRunningTime="2026-07-06 22:28:00.205866798 +0000 UTC m=+15.815868008" Jul 06 22:28:00.206303 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:00.206271 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hjcm5" podStartSLOduration=5.047806907 podStartE2EDuration="15.206263526s" totalImagesPullingTime="10.158456619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:59.262360678 +0000 UTC m=+14.872361889" watchObservedRunningTime="2026-07-06 22:28:00.206263526 +0000 UTC m=+15.816264735" Jul 06 22:28:00.480379 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:00.480282 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:28:00.480907 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:00.480879 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:28:00.997663 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:00.997630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:00.997838 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:00.997630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:00.997838 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:00.997778 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:00.997838 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:00.997798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:00.998003 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:00.997861 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:00.998003 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:00.997934 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:01.192813 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:01.192762 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:28:01.193270 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:01.193243 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9fw9r" Jul 06 22:28:02.545526 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:02.545436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:02.545526 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:02.545486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:02.546137 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.545620 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:28:02.546137 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.545754 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret podName:f20455a2-a971-4b8f-bfab-879716005fbf nodeName:}" failed. No retries permitted until 2026-07-06 22:28:18.545713834 +0000 UTC m=+34.155715023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret") pod "global-pull-secret-syncer-4qqdf" (UID: "f20455a2-a971-4b8f-bfab-879716005fbf") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:28:02.546137 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.545825 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:28:02.546137 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.545848 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:28:02.546137 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.545861 2580 projected.go:196] Error preparing data for projected volume kube-api-access-57jp5 for pod openshift-network-diagnostics/network-check-target-8wq78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:28:02.546137 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.545927 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5 podName:c9a54d7a-954e-405c-a0c5-8fb9a3d37077 nodeName:}" failed. No retries permitted until 2026-07-06 22:28:18.545911295 +0000 UTC m=+34.155912499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-57jp5" (UniqueName: "kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5") pod "network-check-target-8wq78" (UID: "c9a54d7a-954e-405c-a0c5-8fb9a3d37077") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:28:02.646110 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:02.646069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:02.646294 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.646208 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:28:02.646294 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.646266 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs podName:4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433 nodeName:}" failed. No retries permitted until 2026-07-06 22:28:18.646250787 +0000 UTC m=+34.256251978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs") pod "network-metrics-daemon-hcq92" (UID: "4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:28:02.997752 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:02.997712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:02.997752 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:02.997753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:02.997999 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:02.997712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:02.997999 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.997865 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:02.997999 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.997966 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:02.998136 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:02.998019 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:03.198767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:03.198736 2580 generic.go:356] "Generic (PLEG): container finished" podID="37b1c52d-069c-451a-a666-ea8837bf9f7a" containerID="2fd7450b7489b255d3d4e8aba19b2d9c965c499fc554e331040a56e87e00805d" exitCode=0 Jul 06 22:28:03.198935 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:03.198818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerDied","Data":"2fd7450b7489b255d3d4e8aba19b2d9c965c499fc554e331040a56e87e00805d"} Jul 06 22:28:03.291389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:03.291147 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hjcm5_0eca13d8-da9b-48b9-8ce8-f015af526993/dns-node-resolver/0.log" Jul 06 22:28:04.256198 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:04.256155 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t6mx4_e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e/node-ca/0.log" Jul 06 22:28:04.998997 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:04.998954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:04.998997 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:04.998986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:04.999252 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:04.999069 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:04.999252 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:04.999105 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:04.999252 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:04.999145 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:04.999252 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:04.999226 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:06.998076 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:06.998052 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:06.998499 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:06.998086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:06.998499 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:06.998204 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:06.998499 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:06.998295 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:06.998499 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:06.998353 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:06.998499 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:06.998426 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:07.209525 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:07.209490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"d78fe92ad3581a7050b54ba105a9be2c8b18f0d41138ba0d20a9ae32333d668e"} Jul 06 22:28:07.211809 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:07.211780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qmvqb" event={"ID":"2a9a3de8-111a-4398-925e-5d7faa235047","Type":"ContainerStarted","Data":"ac14194ee83410ebe4ff40401376ed58856a16eaf54390ca7c0d2222c8009bae"} Jul 06 22:28:07.335377 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:07.334895 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 22:28:07.870419 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:07.870309 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T22:28:07.335101528Z","UUID":"8666415e-a243-486e-86ec-61d7e180fecb","Handler":null,"Name":"","Endpoint":""} Jul 06 22:28:07.872569 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:07.872547 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 22:28:07.872707 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:07.872592 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 22:28:08.216779 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.216741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" event={"ID":"7f6b6507-63df-4cff-8744-a52377707ff8","Type":"ContainerStarted","Data":"6ab5c94cd7ebfc639a01245a8ed46d041e66b34a71dcbe22173d0c564ef3bec1"} Jul 06 22:28:08.220579 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.220555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"1c4586613f47e9f9be255a181987147002ba9c3d69b7d26ef3ebb3a9e25a9bf6"} Jul 06 22:28:08.220698 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.220617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"b862b098a90f0bfae0d38ee7015130994d170fad28dd2c8d1be44a2ace280353"} Jul 06 22:28:08.220698 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.220648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"ddfff06fbdf9e6e8fd820a44bdf841615b610b9bfd4f6769a4c9612a1a502e4e"} Jul 06 22:28:08.220698 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.220699 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"73c38b3b1374586779c438935f25d07be871903b158815523ac8c2a8706e8bd6"} Jul 06 22:28:08.220857 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.220745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"897f5ec3d59f512e60e1a03dffddb385da6abe35d935767387baf8f472c5821a"} Jul 06 22:28:08.997868 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.997835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:08.997868 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.997851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:08.998189 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:08.997839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:08.998189 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:08.997972 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:08.998189 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:08.998073 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:08.998189 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:08.998175 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:09.224317 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:09.224285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" event={"ID":"7f6b6507-63df-4cff-8744-a52377707ff8","Type":"ContainerStarted","Data":"da9c164eb48b47015bb8b174e55566aedece9c35de06e9b720341dd1df73835c"} Jul 06 22:28:09.242374 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:09.242326 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qmvqb" podStartSLOduration=5.443734509 podStartE2EDuration="24.242312294s" totalImagesPullingTime="18.798577785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:07.233432424 +0000 UTC m=+22.843433634" watchObservedRunningTime="2026-07-06 22:28:09.242312294 +0000 UTC m=+24.852313502" Jul 06 22:28:09.242704 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:09.242684 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjd9" podStartSLOduration=4.500107317 podStartE2EDuration="24.242678551s" totalImagesPullingTime="19.742571234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:09.241237594 +0000 UTC m=+24.851238802" watchObservedRunningTime="2026-07-06 22:28:09.242678551 +0000 UTC m=+24.852679760" Jul 06 22:28:10.997587 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:10.997549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:10.998208 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:10.997549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:10.998208 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:10.997689 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:10.998208 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:10.997549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:10.998208 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:10.997753 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:10.998208 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:10.997859 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:11.233038 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:11.233002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"791d78e889fe859c528244aadb211248aa8123ba6b9415a358e46f2ceccb1943"} Jul 06 22:28:12.236972 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:12.236940 2580 generic.go:356] "Generic (PLEG): container finished" podID="37b1c52d-069c-451a-a666-ea8837bf9f7a" containerID="cc4279d143755ece418fc3066da86a7181531c3149b04d2be34286d935abbf5a" exitCode=0 Jul 06 22:28:12.237390 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:12.236999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerDied","Data":"cc4279d143755ece418fc3066da86a7181531c3149b04d2be34286d935abbf5a"} Jul 06 22:28:12.998241 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:12.998201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:12.998241 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:12.998216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:12.998446 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:12.998199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:12.998446 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:12.998313 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:12.998446 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:12.998404 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:12.998542 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:12.998469 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:14.244532 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.244141 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" event={"ID":"b0dc89e4-c797-4a01-bb1d-43b919feba1e","Type":"ContainerStarted","Data":"60bd0534fdb0239336f2596a1ec229cf794bca981e0c93b7cf4a4ef000b785e0"} Jul 06 22:28:14.244532 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.244419 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:28:14.244532 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.244471 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:28:14.246415 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.246387 2580 generic.go:356] "Generic (PLEG): container finished" podID="37b1c52d-069c-451a-a666-ea8837bf9f7a" containerID="2f8af2c9ce17918571bf2d3bb4f1902305488a3a34a9128f4fe634b96bf0035c" exitCode=0 Jul 06 22:28:14.246535 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.246425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerDied","Data":"2f8af2c9ce17918571bf2d3bb4f1902305488a3a34a9128f4fe634b96bf0035c"} Jul 06 22:28:14.260610 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.260589 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:28:14.273986 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.273945 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" podStartSLOduration=10.423201961 podStartE2EDuration="29.273931041s" totalImagesPullingTime="18.85072908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:14.273800605 +0000 UTC m=+29.883801814" watchObservedRunningTime="2026-07-06 22:28:14.273931041 +0000 UTC m=+29.883932249" Jul 06 22:28:14.999463 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.999372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:14.999680 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.999521 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:14.999680 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:14.999537 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:14.999680 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:14.999594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:14.999840 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:14.999698 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:14.999840 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:14.999782 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:15.250591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:15.250448 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:28:15.276398 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:15.276367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:28:15.498040 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:15.497797 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8wq78"] Jul 06 22:28:15.498282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:15.498138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:15.498377 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:15.498271 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:15.498761 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:15.498740 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4qqdf"] Jul 06 22:28:15.498887 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:15.498829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:15.498962 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:15.498918 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:15.499519 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:15.499494 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hcq92"] Jul 06 22:28:15.499635 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:15.499588 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:15.499713 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:15.499687 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:16.254345 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:16.254312 2580 generic.go:356] "Generic (PLEG): container finished" podID="37b1c52d-069c-451a-a666-ea8837bf9f7a" containerID="f9a10e4a7034108a2424399692fdd0d6fef77603265698efa3d9dec8130ab438" exitCode=0 Jul 06 22:28:16.254844 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:16.254383 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerDied","Data":"f9a10e4a7034108a2424399692fdd0d6fef77603265698efa3d9dec8130ab438"} Jul 06 22:28:16.997627 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:16.997576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:16.997822 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:16.997576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:16.997822 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:16.997723 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:16.997822 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:16.997576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:16.997822 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:16.997795 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:16.998024 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:16.997857 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:18.568491 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:18.568457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:18.568491 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:18.568499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:18.569221 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.568624 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:28:18.569221 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.568646 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:28:18.569221 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.568661 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:28:18.569221 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.568672 2580 projected.go:196] Error preparing data for projected volume kube-api-access-57jp5 for pod openshift-network-diagnostics/network-check-target-8wq78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:28:18.569221 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.568703 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret podName:f20455a2-a971-4b8f-bfab-879716005fbf nodeName:}" failed. No retries permitted until 2026-07-06 22:28:50.568687229 +0000 UTC m=+66.178688439 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret") pod "global-pull-secret-syncer-4qqdf" (UID: "f20455a2-a971-4b8f-bfab-879716005fbf") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:28:18.569221 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.568718 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5 podName:c9a54d7a-954e-405c-a0c5-8fb9a3d37077 nodeName:}" failed. No retries permitted until 2026-07-06 22:28:50.568712723 +0000 UTC m=+66.178713909 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-57jp5" (UniqueName: "kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5") pod "network-check-target-8wq78" (UID: "c9a54d7a-954e-405c-a0c5-8fb9a3d37077") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:28:18.669484 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:18.669452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:18.669668 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.669591 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:28:18.669668 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.669646 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs podName:4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433 nodeName:}" failed. No retries permitted until 2026-07-06 22:28:50.669629895 +0000 UTC m=+66.279631082 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs") pod "network-metrics-daemon-hcq92" (UID: "4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:28:18.998234 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:18.998199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:18.998415 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:18.998199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:18.998415 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:18.998210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:18.998415 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.998350 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcq92" podUID="4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433" Jul 06 22:28:18.998415 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.998409 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wq78" podUID="c9a54d7a-954e-405c-a0c5-8fb9a3d37077" Jul 06 22:28:18.998636 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:18.998522 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqdf" podUID="f20455a2-a971-4b8f-bfab-879716005fbf" Jul 06 22:28:19.564318 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:19.564286 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-56.ec2.internal" event="NodeReady" Jul 06 22:28:19.564468 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:19.564415 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 22:28:20.997832 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:20.997795 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:20.997832 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:20.997813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:20.998573 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:20.997795 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:21.000619 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:21.000588 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:28:21.001905 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:21.001880 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:28:21.002033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:21.001885 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:28:21.002033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:21.001888 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:28:21.002157 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:21.002071 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-69kbf\"" Jul 06 22:28:21.002248 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:21.002233 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kgqgb\"" Jul 06 22:28:21.271632 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:21.271604 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerStarted","Data":"1b2d43f44077a0f784be335f9eb765bce6d4b7c8baf5c7492e89fc890ede747d"} Jul 06 22:28:22.276882 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:22.276851 2580 generic.go:356] "Generic (PLEG): container finished" podID="37b1c52d-069c-451a-a666-ea8837bf9f7a" containerID="1b2d43f44077a0f784be335f9eb765bce6d4b7c8baf5c7492e89fc890ede747d" exitCode=0 Jul 06 22:28:22.277314 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:22.276920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerDied","Data":"1b2d43f44077a0f784be335f9eb765bce6d4b7c8baf5c7492e89fc890ede747d"} Jul 06 22:28:23.281906 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:23.281874 2580 generic.go:356] "Generic (PLEG): container finished" podID="37b1c52d-069c-451a-a666-ea8837bf9f7a" containerID="0d3bff826bf4fbdb3e289f66d9894a42e1099a8df1540346cafa5816588fd2b1" exitCode=0 Jul 06 22:28:23.282389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:23.281931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerDied","Data":"0d3bff826bf4fbdb3e289f66d9894a42e1099a8df1540346cafa5816588fd2b1"} Jul 06 22:28:24.287427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:24.287396 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" event={"ID":"37b1c52d-069c-451a-a666-ea8837bf9f7a","Type":"ContainerStarted","Data":"03f1e9cc12587940b048a86822cf43c753dece56b3a857bb4f87ace613c6d25c"} Jul 06 22:28:24.314022 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:24.313975 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pwl2c" podStartSLOduration=13.905040336999999 podStartE2EDuration="39.313960039s" totalImagesPullingTime="25.408919702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:24.312532984 +0000 UTC m=+39.922534196" watchObservedRunningTime="2026-07-06 22:28:24.313960039 +0000 UTC m=+39.923961247" Jul 06 22:28:39.297642 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.297472 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8qbqw"] Jul 06 22:28:39.341044 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.341014 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vzb48"] Jul 06 22:28:39.341213 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.341140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.344510 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.344489 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 22:28:39.344799 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.344782 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 22:28:39.344851 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.344835 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z9fkj\"" Jul 06 22:28:39.360857 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.360837 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8qbqw"] Jul 06 22:28:39.360971 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.360869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vzb48"] Jul 06 22:28:39.360971 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.360944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vzb48" Jul 06 22:28:39.363791 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.363772 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 22:28:39.363910 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.363793 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 22:28:39.363910 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.363870 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 22:28:39.364065 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.363980 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mvrsx\"" Jul 06 22:28:39.388827 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.388801 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w6g5c"] Jul 06 22:28:39.401172 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.401145 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.402794 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.402776 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w6g5c"] Jul 06 22:28:39.403650 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.403615 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 22:28:39.403746 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.403651 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 22:28:39.403746 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.403658 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 22:28:39.403847 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.403796 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mcxxp\"" Jul 06 22:28:39.403847 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.403805 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 22:28:39.519296 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdr98\" (UniqueName: \"kubernetes.io/projected/f618256d-c844-4d04-9f8f-0c341470f10c-kube-api-access-rdr98\") pod \"ingress-canary-vzb48\" (UID: \"f618256d-c844-4d04-9f8f-0c341470f10c\") " pod="openshift-ingress-canary/ingress-canary-vzb48" Jul 06 22:28:39.519296 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519301 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.519534 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czp9h\" (UniqueName: \"kubernetes.io/projected/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-kube-api-access-czp9h\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.519534 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4be3f479-cdd8-4f7f-993f-4fa10aac623e-config-volume\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.519534 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.519534 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4be3f479-cdd8-4f7f-993f-4fa10aac623e-metrics-tls\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.519534 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-crio-socket\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.519534 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6hc\" (UniqueName: \"kubernetes.io/projected/4be3f479-cdd8-4f7f-993f-4fa10aac623e-kube-api-access-4t6hc\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.519735 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-data-volume\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.519735 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f618256d-c844-4d04-9f8f-0c341470f10c-cert\") pod \"ingress-canary-vzb48\" (UID: \"f618256d-c844-4d04-9f8f-0c341470f10c\") " pod="openshift-ingress-canary/ingress-canary-vzb48" Jul 06 22:28:39.519735 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.519689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4be3f479-cdd8-4f7f-993f-4fa10aac623e-tmp-dir\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.620892 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.620809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdr98\" (UniqueName: \"kubernetes.io/projected/f618256d-c844-4d04-9f8f-0c341470f10c-kube-api-access-rdr98\") pod \"ingress-canary-vzb48\" (UID: \"f618256d-c844-4d04-9f8f-0c341470f10c\") " pod="openshift-ingress-canary/ingress-canary-vzb48" Jul 06 22:28:39.620892 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.620840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.620892 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.620858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czp9h\" (UniqueName: \"kubernetes.io/projected/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-kube-api-access-czp9h\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.620892 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.620874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4be3f479-cdd8-4f7f-993f-4fa10aac623e-config-volume\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.620892 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.620891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.621280 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4be3f479-cdd8-4f7f-993f-4fa10aac623e-metrics-tls\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.621280 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-crio-socket\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.621280 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6hc\" (UniqueName: \"kubernetes.io/projected/4be3f479-cdd8-4f7f-993f-4fa10aac623e-kube-api-access-4t6hc\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.621280 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-data-volume\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.621280 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621242 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-crio-socket\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.621280 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f618256d-c844-4d04-9f8f-0c341470f10c-cert\") pod \"ingress-canary-vzb48\" (UID: \"f618256d-c844-4d04-9f8f-0c341470f10c\") " pod="openshift-ingress-canary/ingress-canary-vzb48" Jul 06 22:28:39.621497 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4be3f479-cdd8-4f7f-993f-4fa10aac623e-tmp-dir\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.621591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621568 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4be3f479-cdd8-4f7f-993f-4fa10aac623e-config-volume\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.621635 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621579 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.621635 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.621622 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-data-volume\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.624902 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.624877 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.625005 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.624918 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f618256d-c844-4d04-9f8f-0c341470f10c-cert\") pod \"ingress-canary-vzb48\" (UID: \"f618256d-c844-4d04-9f8f-0c341470f10c\") " pod="openshift-ingress-canary/ingress-canary-vzb48" Jul 06 22:28:39.629147 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.629124 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czp9h\" (UniqueName: \"kubernetes.io/projected/4fd1bf24-9a45-494f-971c-f67d4ab7d3c5-kube-api-access-czp9h\") pod \"insights-runtime-extractor-w6g5c\" (UID: \"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5\") " pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.629559 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.629539 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdr98\" (UniqueName: \"kubernetes.io/projected/f618256d-c844-4d04-9f8f-0c341470f10c-kube-api-access-rdr98\") pod \"ingress-canary-vzb48\" (UID: \"f618256d-c844-4d04-9f8f-0c341470f10c\") " pod="openshift-ingress-canary/ingress-canary-vzb48" Jul 06 22:28:39.636623 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.636599 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4be3f479-cdd8-4f7f-993f-4fa10aac623e-tmp-dir\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.638189 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.638153 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4be3f479-cdd8-4f7f-993f-4fa10aac623e-metrics-tls\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.638311 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.638293 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6hc\" (UniqueName: \"kubernetes.io/projected/4be3f479-cdd8-4f7f-993f-4fa10aac623e-kube-api-access-4t6hc\") pod \"dns-default-8qbqw\" (UID: \"4be3f479-cdd8-4f7f-993f-4fa10aac623e\") " pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.650230 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.650206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:39.669322 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.669303 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vzb48" Jul 06 22:28:39.713033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.712766 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w6g5c" Jul 06 22:28:39.868709 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.868675 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vzb48"] Jul 06 22:28:39.871385 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.871336 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8qbqw"] Jul 06 22:28:39.872352 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:39.872243 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf618256d_c844_4d04_9f8f_0c341470f10c.slice/crio-fce508126975f6b76768a4b22c0310ccd1c9916919ab98e2d2cdfb80debe9842 WatchSource:0}: Error finding container fce508126975f6b76768a4b22c0310ccd1c9916919ab98e2d2cdfb80debe9842: Status 404 returned error can't find the container with id fce508126975f6b76768a4b22c0310ccd1c9916919ab98e2d2cdfb80debe9842 Jul 06 22:28:39.875947 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:39.875905 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be3f479_cdd8_4f7f_993f_4fa10aac623e.slice/crio-35d1d188fd65573f0e0a457a0156fd7bdbad4f168e450cfceff05ae01efc61a5 WatchSource:0}: Error finding container 35d1d188fd65573f0e0a457a0156fd7bdbad4f168e450cfceff05ae01efc61a5: Status 404 returned error can't find the container with id 35d1d188fd65573f0e0a457a0156fd7bdbad4f168e450cfceff05ae01efc61a5 Jul 06 22:28:39.888550 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:39.888527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w6g5c"] Jul 06 22:28:39.891705 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:39.891669 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd1bf24_9a45_494f_971c_f67d4ab7d3c5.slice/crio-092e1b9833877537be513feea14b774574b228a03ccc431fb7a27a500f4f937d WatchSource:0}: Error finding container 092e1b9833877537be513feea14b774574b228a03ccc431fb7a27a500f4f937d: Status 404 returned error can't find the container with id 092e1b9833877537be513feea14b774574b228a03ccc431fb7a27a500f4f937d Jul 06 22:28:40.324831 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:40.324790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qbqw" event={"ID":"4be3f479-cdd8-4f7f-993f-4fa10aac623e","Type":"ContainerStarted","Data":"35d1d188fd65573f0e0a457a0156fd7bdbad4f168e450cfceff05ae01efc61a5"} Jul 06 22:28:40.326028 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:40.325999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6g5c" event={"ID":"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5","Type":"ContainerStarted","Data":"092e1b9833877537be513feea14b774574b228a03ccc431fb7a27a500f4f937d"} Jul 06 22:28:40.327203 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:40.327158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vzb48" event={"ID":"f618256d-c844-4d04-9f8f-0c341470f10c","Type":"ContainerStarted","Data":"fce508126975f6b76768a4b22c0310ccd1c9916919ab98e2d2cdfb80debe9842"} Jul 06 22:28:44.472123 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.472090 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw"] Jul 06 22:28:44.491514 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.491490 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw"] Jul 06 22:28:44.491644 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.491585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.496069 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.496033 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 22:28:44.496223 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.496076 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 22:28:44.496223 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.496114 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 22:28:44.496223 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.496043 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ffh7z\"" Jul 06 22:28:44.496223 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.496074 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 22:28:44.496223 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.496180 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 22:28:44.554469 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.554437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.554469 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.554468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e32295-7644-416c-abcb-c39454bea80b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.554645 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.554497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.554645 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.554544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blsz\" (UniqueName: \"kubernetes.io/projected/50e32295-7644-416c-abcb-c39454bea80b-kube-api-access-2blsz\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.655177 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.655120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.655177 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.655177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e32295-7644-416c-abcb-c39454bea80b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.655398 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.655226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.655398 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.655244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blsz\" (UniqueName: \"kubernetes.io/projected/50e32295-7644-416c-abcb-c39454bea80b-kube-api-access-2blsz\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.655398 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:44.655300 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 22:28:44.655398 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:44.655385 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-tls podName:50e32295-7644-416c-abcb-c39454bea80b nodeName:}" failed. No retries permitted until 2026-07-06 22:28:45.155366499 +0000 UTC m=+60.765367686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-dkcnw" (UID: "50e32295-7644-416c-abcb-c39454bea80b") : secret "prometheus-operator-tls" not found Jul 06 22:28:44.655937 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.655912 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e32295-7644-416c-abcb-c39454bea80b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.657911 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.657886 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:44.665074 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:44.665050 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blsz\" (UniqueName: \"kubernetes.io/projected/50e32295-7644-416c-abcb-c39454bea80b-kube-api-access-2blsz\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:45.158427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:45.158393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:45.160711 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:45.160684 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50e32295-7644-416c-abcb-c39454bea80b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dkcnw\" (UID: \"50e32295-7644-416c-abcb-c39454bea80b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:45.403526 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:45.403494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ffh7z\"" Jul 06 22:28:45.411707 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:45.411612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" Jul 06 22:28:45.593177 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:45.593084 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw"] Jul 06 22:28:45.598362 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:45.598325 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e32295_7644_416c_abcb_c39454bea80b.slice/crio-fec857f0170623b2ae068befa3d450accabfafbcd598495d0bd1a1c52583a79a WatchSource:0}: Error finding container fec857f0170623b2ae068befa3d450accabfafbcd598495d0bd1a1c52583a79a: Status 404 returned error can't find the container with id fec857f0170623b2ae068befa3d450accabfafbcd598495d0bd1a1c52583a79a Jul 06 22:28:46.345292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:46.345202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6g5c" event={"ID":"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5","Type":"ContainerStarted","Data":"0032f280a98225ef2ef11f1d2c975bdbcd48185984a2a4f7d3059af2873d471d"} Jul 06 22:28:46.345451 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:46.345315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6g5c" event={"ID":"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5","Type":"ContainerStarted","Data":"a6de43b196d7e8951b4d560491d34efc73f7c6584f2fb281d2096059e5420e95"} Jul 06 22:28:46.346802 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:46.346745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vzb48" event={"ID":"f618256d-c844-4d04-9f8f-0c341470f10c","Type":"ContainerStarted","Data":"071fb99ae383fa533688b8b6b7b4a4c8e6ce6f336d0767380331735adbacbf5b"} Jul 06 22:28:46.348083 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:46.348059 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" event={"ID":"50e32295-7644-416c-abcb-c39454bea80b","Type":"ContainerStarted","Data":"fec857f0170623b2ae068befa3d450accabfafbcd598495d0bd1a1c52583a79a"} Jul 06 22:28:46.350189 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:46.350073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qbqw" event={"ID":"4be3f479-cdd8-4f7f-993f-4fa10aac623e","Type":"ContainerStarted","Data":"c42e096c9d7faa83d27cad09e6228688a3b92e2a5c867b16da0e3491f4ddd43b"} Jul 06 22:28:46.350290 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:46.350214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qbqw" event={"ID":"4be3f479-cdd8-4f7f-993f-4fa10aac623e","Type":"ContainerStarted","Data":"c54c3fa1684178fdfea5c855a34ea80f5b7fd90891756d2bc3d71e2bf7515b63"} Jul 06 22:28:46.350290 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:46.350284 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:46.363313 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:46.363263 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vzb48" podStartSLOduration=1.775360721 podStartE2EDuration="7.363247465s" totalImagesPullingTime="5.587886744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:46.362667743 +0000 UTC m=+61.972668978" watchObservedRunningTime="2026-07-06 22:28:46.363247465 +0000 UTC m=+61.973248675" Jul 06 22:28:47.268812 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:47.268740 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqkmg" Jul 06 22:28:47.299742 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:47.299681 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8qbqw" podStartSLOduration=2.729726872 podStartE2EDuration="8.299658862s" totalImagesPullingTime="5.56993199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:46.379989378 +0000 UTC m=+61.989990586" watchObservedRunningTime="2026-07-06 22:28:47.299658862 +0000 UTC m=+62.909660074" Jul 06 22:28:47.356006 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:47.355956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" event={"ID":"50e32295-7644-416c-abcb-c39454bea80b","Type":"ContainerStarted","Data":"9e80d89ea600cf187efc457f23fe24989bef916628b703530dfbb97a0c6f7254"} Jul 06 22:28:47.356196 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:47.356021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" event={"ID":"50e32295-7644-416c-abcb-c39454bea80b","Type":"ContainerStarted","Data":"d6d2e6259872386db96c7eb1c6336940b7af1e5ff413b7c92a121c7e49ab4e42"} Jul 06 22:28:47.376342 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:47.376242 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dkcnw" podStartSLOduration=2.013837579 podStartE2EDuration="3.376157335s" totalImagesPullingTime="1.362319756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:47.375342713 +0000 UTC m=+62.985343950" watchObservedRunningTime="2026-07-06 22:28:47.376157335 +0000 UTC m=+62.986158547" Jul 06 22:28:48.826431 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.826350 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl"] Jul 06 22:28:48.830814 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.830789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.833406 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.833383 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 22:28:48.833574 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.833382 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 22:28:48.833658 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.833644 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t7ngf\"" Jul 06 22:28:48.842995 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.842972 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl"] Jul 06 22:28:48.847038 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.847014 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj"] Jul 06 22:28:48.849010 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.848994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.851713 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.851681 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 22:28:48.851713 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.851706 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 22:28:48.851879 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.851689 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fjs49\"" Jul 06 22:28:48.851879 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.851728 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 22:28:48.856905 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.856886 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-94cww"] Jul 06 22:28:48.858765 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.858749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.860316 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.860298 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj"] Jul 06 22:28:48.861142 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.861122 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 22:28:48.861387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.861372 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 22:28:48.861505 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.861490 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kml7g\"" Jul 06 22:28:48.861505 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.861495 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 22:28:48.883382 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883495 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.883495 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxp4\" (UniqueName: \"kubernetes.io/projected/54117083-55de-423b-8b4b-7c9e01649591-kube-api-access-5hxp4\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883495 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.883495 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gv2x\" (UniqueName: \"kubernetes.io/projected/110b2bb5-1ae2-4d9e-8488-0ae89109feae-kube-api-access-7gv2x\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.883495 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-textfile\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883650 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-accelerators-collector-config\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883650 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883556 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.883650 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-wtmp\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883650 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.883650 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.883794 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-root\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883794 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.883794 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-sys\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883794 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54117083-55de-423b-8b4b-7c9e01649591-metrics-client-ca\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883794 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492n4\" (UniqueName: \"kubernetes.io/projected/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-api-access-492n4\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.883794 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-tls\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.883970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.883970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.883839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/110b2bb5-1ae2-4d9e-8488-0ae89109feae-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.985079 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-wtmp\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985079 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.985292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.985292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-root\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985292 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:48.985227 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 22:28:48.985292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.985292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985253 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-wtmp\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-sys\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985282 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-root\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985292 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:48.985296 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-tls podName:110b2bb5-1ae2-4d9e-8488-0ae89109feae nodeName:}" failed. No retries permitted until 2026-07-06 22:28:49.485277153 +0000 UTC m=+65.095278355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-t5txl" (UID: "110b2bb5-1ae2-4d9e-8488-0ae89109feae") : secret "openshift-state-metrics-tls" not found Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985308 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54117083-55de-423b-8b4b-7c9e01649591-sys\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54117083-55de-423b-8b4b-7c9e01649591-metrics-client-ca\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:48.985356 2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492n4\" (UniqueName: \"kubernetes.io/projected/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-api-access-492n4\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:48.985392 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-tls podName:f0155679-45f4-4ca6-addd-f9cf5b85d1bd nodeName:}" failed. No retries permitted until 2026-07-06 22:28:49.485380233 +0000 UTC m=+65.095381424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-6q2pj" (UID: "f0155679-45f4-4ca6-addd-f9cf5b85d1bd") : secret "kube-state-metrics-tls" not found Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-tls\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/110b2bb5-1ae2-4d9e-8488-0ae89109feae-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985503 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:48.985558 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxp4\" (UniqueName: \"kubernetes.io/projected/54117083-55de-423b-8b4b-7c9e01649591-kube-api-access-5hxp4\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.985621 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:28:48.985623 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-tls podName:54117083-55de-423b-8b4b-7c9e01649591 nodeName:}" failed. No retries permitted until 2026-07-06 22:28:49.485606007 +0000 UTC m=+65.095607218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-tls") pod "node-exporter-94cww" (UID: "54117083-55de-423b-8b4b-7c9e01649591") : secret "node-exporter-tls" not found Jul 06 22:28:48.986364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gv2x\" (UniqueName: \"kubernetes.io/projected/110b2bb5-1ae2-4d9e-8488-0ae89109feae-kube-api-access-7gv2x\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.986364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-textfile\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.986364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.985954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-accelerators-collector-config\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.986364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.986000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.986364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.986102 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54117083-55de-423b-8b4b-7c9e01649591-metrics-client-ca\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.986364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.986301 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/110b2bb5-1ae2-4d9e-8488-0ae89109feae-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.986364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.986309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.986693 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.986456 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-textfile\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.986767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.986740 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-accelerators-collector-config\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.986848 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.986827 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.987942 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.987912 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.988062 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.987995 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.988062 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.988025 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:48.993971 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.993945 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxp4\" (UniqueName: \"kubernetes.io/projected/54117083-55de-423b-8b4b-7c9e01649591-kube-api-access-5hxp4\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:48.994200 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.994179 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-492n4\" (UniqueName: \"kubernetes.io/projected/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-api-access-492n4\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:48.995209 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:48.995192 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gv2x\" (UniqueName: \"kubernetes.io/projected/110b2bb5-1ae2-4d9e-8488-0ae89109feae-kube-api-access-7gv2x\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:49.366394 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.366357 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6g5c" event={"ID":"4fd1bf24-9a45-494f-971c-f67d4ab7d3c5","Type":"ContainerStarted","Data":"e8e8a242760fe6fe3167b6d1b39fe869ce66ea82bffd1406191b5a131cf61f5d"} Jul 06 22:28:49.388889 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.388834 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w6g5c" podStartSLOduration=1.950694361 podStartE2EDuration="10.388820416s" totalImagesPullingTime="8.438126055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:28:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:49.38852811 +0000 UTC m=+64.998529320" watchObservedRunningTime="2026-07-06 22:28:49.388820416 +0000 UTC m=+64.998821623" Jul 06 22:28:49.490811 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.490775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:49.490811 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.490813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:49.491080 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.490841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-tls\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:49.493150 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.493124 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54117083-55de-423b-8b4b-7c9e01649591-node-exporter-tls\") pod \"node-exporter-94cww\" (UID: \"54117083-55de-423b-8b4b-7c9e01649591\") " pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:49.493348 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.493266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/110b2bb5-1ae2-4d9e-8488-0ae89109feae-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-t5txl\" (UID: \"110b2bb5-1ae2-4d9e-8488-0ae89109feae\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:49.493406 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.493347 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0155679-45f4-4ca6-addd-f9cf5b85d1bd-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6q2pj\" (UID: \"f0155679-45f4-4ca6-addd-f9cf5b85d1bd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:49.739613 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.739582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" Jul 06 22:28:49.757391 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.757366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" Jul 06 22:28:49.767569 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.767546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-94cww" Jul 06 22:28:49.883884 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.883786 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl"] Jul 06 22:28:49.886591 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:49.886568 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110b2bb5_1ae2_4d9e_8488_0ae89109feae.slice/crio-5eff4712e5b6d8c04c00ca63492dcb68cbf54c1765a14a6369c860eae6f16cd1 WatchSource:0}: Error finding container 5eff4712e5b6d8c04c00ca63492dcb68cbf54c1765a14a6369c860eae6f16cd1: Status 404 returned error can't find the container with id 5eff4712e5b6d8c04c00ca63492dcb68cbf54c1765a14a6369c860eae6f16cd1 Jul 06 22:28:49.902107 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.902079 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj"] Jul 06 22:28:49.905102 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:49.905064 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0155679_45f4_4ca6_addd_f9cf5b85d1bd.slice/crio-b54e121ce086c8eb9f3f484c1839817c5dab10fcd8e4795b9c4c463c5627d408 WatchSource:0}: Error finding container b54e121ce086c8eb9f3f484c1839817c5dab10fcd8e4795b9c4c463c5627d408: Status 404 returned error can't find the container with id b54e121ce086c8eb9f3f484c1839817c5dab10fcd8e4795b9c4c463c5627d408 Jul 06 22:28:49.929376 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.929349 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:28:49.937626 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.937596 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.940522 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.940503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-22pmh\"" Jul 06 22:28:49.940522 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.940503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 22:28:49.940522 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.940500 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 22:28:49.940772 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.940753 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 22:28:49.940829 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.940786 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 22:28:49.940940 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.940924 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 22:28:49.941371 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.941225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 22:28:49.941371 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.941242 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 22:28:49.941371 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.941260 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 22:28:49.941371 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.941266 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 22:28:49.948203 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.948183 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:28:49.995342 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995421 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995421 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995421 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995539 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995539 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-volume\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995605 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995605 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97c5n\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-kube-api-access-97c5n\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995681 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995718 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-web-config\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995762 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995798 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:49.995798 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:49.995778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-out\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.096943 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.096912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.096943 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.096945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97c5n\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-kube-api-access-97c5n\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.096966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.096989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-web-config\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-out\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.097591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-volume\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.098188 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.097840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.098188 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.098032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.098355 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.098333 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.100318 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.100273 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-web-config\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.100488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.100443 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.100488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.100458 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-volume\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.100605 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.100579 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.100782 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.100756 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.101106 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.101084 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.101146 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.101092 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.101442 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.101422 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-out\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.101653 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.101637 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.105096 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.105073 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97c5n\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-kube-api-access-97c5n\") pod \"alertmanager-main-0\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.248901 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.248813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:28:50.373831 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.373796 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94cww" event={"ID":"54117083-55de-423b-8b4b-7c9e01649591","Type":"ContainerStarted","Data":"22e07ed48ebf3c77b136905e187830eb878423f94dcdc423c3a8cc701a401de3"} Jul 06 22:28:50.376202 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.376148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" event={"ID":"110b2bb5-1ae2-4d9e-8488-0ae89109feae","Type":"ContainerStarted","Data":"edc6dac6f74a1980e0e05039edd78410a40b925e777f8cfda366734f9fb9c8e7"} Jul 06 22:28:50.376327 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.376233 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" event={"ID":"110b2bb5-1ae2-4d9e-8488-0ae89109feae","Type":"ContainerStarted","Data":"fea65db8d350c5ac9f10102c24040db4666736dda6a3a5f4e82e50ef3a2545d5"} Jul 06 22:28:50.376327 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.376264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" event={"ID":"110b2bb5-1ae2-4d9e-8488-0ae89109feae","Type":"ContainerStarted","Data":"5eff4712e5b6d8c04c00ca63492dcb68cbf54c1765a14a6369c860eae6f16cd1"} Jul 06 22:28:50.377448 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.377412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" event={"ID":"f0155679-45f4-4ca6-addd-f9cf5b85d1bd","Type":"ContainerStarted","Data":"b54e121ce086c8eb9f3f484c1839817c5dab10fcd8e4795b9c4c463c5627d408"} Jul 06 22:28:50.405097 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.405061 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:28:50.410895 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:50.410863 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46549a4_1a24_4c36_8eb6_e93a26e8ec44.slice/crio-060208148e7b3984e47c1b15cf933eb6f52e350251c38d261d6e61da955c595e WatchSource:0}: Error finding container 060208148e7b3984e47c1b15cf933eb6f52e350251c38d261d6e61da955c595e: Status 404 returned error can't find the container with id 060208148e7b3984e47c1b15cf933eb6f52e350251c38d261d6e61da955c595e Jul 06 22:28:50.600264 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.600181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:50.600417 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.600305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:50.603035 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.603009 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:28:50.603201 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.603054 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:28:50.613853 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.613811 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f20455a2-a971-4b8f-bfab-879716005fbf-original-pull-secret\") pod \"global-pull-secret-syncer-4qqdf\" (UID: \"f20455a2-a971-4b8f-bfab-879716005fbf\") " pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:50.613973 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.613938 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:28:50.624634 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.624582 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/c9a54d7a-954e-405c-a0c5-8fb9a3d37077-kube-api-access-57jp5\") pod \"network-check-target-8wq78\" (UID: \"c9a54d7a-954e-405c-a0c5-8fb9a3d37077\") " pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:50.701520 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.701484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:50.704405 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.704346 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:28:50.711545 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.711515 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqdf" Jul 06 22:28:50.715906 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.715853 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433-metrics-certs\") pod \"network-metrics-daemon-hcq92\" (UID: \"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433\") " pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:50.722076 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.722057 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-69kbf\"" Jul 06 22:28:50.726350 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.726298 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kgqgb\"" Jul 06 22:28:50.729354 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.729325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:50.734391 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.734358 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcq92" Jul 06 22:28:50.936899 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.936810 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hcq92"] Jul 06 22:28:50.943340 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.942614 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7985b97cb5-5whbv"] Jul 06 22:28:50.947360 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.946298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:50.950876 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.949187 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 22:28:50.950876 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.949199 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kj6z5\"" Jul 06 22:28:50.951915 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.951302 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 22:28:50.951915 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.951315 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 22:28:50.951915 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.951550 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 22:28:50.951915 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.951589 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-laut53sd83ku\"" Jul 06 22:28:50.951915 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.951594 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 22:28:50.958054 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.957970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7985b97cb5-5whbv"] Jul 06 22:28:50.962957 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.961233 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4qqdf"] Jul 06 22:28:50.962957 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:50.962669 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8wq78"] Jul 06 22:28:50.967724 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:50.967657 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20455a2_a971_4b8f_bfab_879716005fbf.slice/crio-e097a64fa8bb4a574dc49af37e9773cae4da6107203a1dcc49f0cb308fcc5c98 WatchSource:0}: Error finding container e097a64fa8bb4a574dc49af37e9773cae4da6107203a1dcc49f0cb308fcc5c98: Status 404 returned error can't find the container with id e097a64fa8bb4a574dc49af37e9773cae4da6107203a1dcc49f0cb308fcc5c98 Jul 06 22:28:50.968350 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:50.968323 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a54d7a_954e_405c_a0c5_8fb9a3d37077.slice/crio-4746060b6a5b6d61ddcf5bffcc0d156be22a6d41f2d4eb2b1b0f081a6743205f WatchSource:0}: Error finding container 4746060b6a5b6d61ddcf5bffcc0d156be22a6d41f2d4eb2b1b0f081a6743205f: Status 404 returned error can't find the container with id 4746060b6a5b6d61ddcf5bffcc0d156be22a6d41f2d4eb2b1b0f081a6743205f Jul 06 22:28:51.004533 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.004479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.004533 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.004532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmvc\" (UniqueName: \"kubernetes.io/projected/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-kube-api-access-8kmvc\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.004768 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.004599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.004768 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.004670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.004768 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.004732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-metrics-client-ca\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.004883 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.004777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.004883 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.004812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-grpc-tls\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.004883 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.004845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-tls\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.105941 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.105857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-metrics-client-ca\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.105941 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.105913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.106191 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.105958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-grpc-tls\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.106191 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.106000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-tls\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.106191 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.106143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.106335 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.106211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmvc\" (UniqueName: \"kubernetes.io/projected/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-kube-api-access-8kmvc\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.106335 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.106243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.106335 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.106303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.106722 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.106696 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-metrics-client-ca\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.109083 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.109060 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-grpc-tls\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.109083 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.109073 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.109264 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.109233 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-tls\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.109447 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.109416 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.109676 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.109650 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.109747 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.109722 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.114648 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.114629 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmvc\" (UniqueName: \"kubernetes.io/projected/d95494d6-fb6b-428d-b9cd-a9411a4ec95f-kube-api-access-8kmvc\") pod \"thanos-querier-7985b97cb5-5whbv\" (UID: \"d95494d6-fb6b-428d-b9cd-a9411a4ec95f\") " pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.264853 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.264813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:28:51.382839 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.382748 2580 generic.go:356] "Generic (PLEG): container finished" podID="54117083-55de-423b-8b4b-7c9e01649591" containerID="c4f03f2ce55f17979197b4fe2c04e356024d0d53f6ee586c25ba90198b646d62" exitCode=0 Jul 06 22:28:51.383008 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.382838 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94cww" event={"ID":"54117083-55de-423b-8b4b-7c9e01649591","Type":"ContainerDied","Data":"c4f03f2ce55f17979197b4fe2c04e356024d0d53f6ee586c25ba90198b646d62"} Jul 06 22:28:51.384229 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.384200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerStarted","Data":"060208148e7b3984e47c1b15cf933eb6f52e350251c38d261d6e61da955c595e"} Jul 06 22:28:51.385405 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.385380 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4qqdf" event={"ID":"f20455a2-a971-4b8f-bfab-879716005fbf","Type":"ContainerStarted","Data":"e097a64fa8bb4a574dc49af37e9773cae4da6107203a1dcc49f0cb308fcc5c98"} Jul 06 22:28:51.386709 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.386684 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8wq78" event={"ID":"c9a54d7a-954e-405c-a0c5-8fb9a3d37077","Type":"ContainerStarted","Data":"4746060b6a5b6d61ddcf5bffcc0d156be22a6d41f2d4eb2b1b0f081a6743205f"} Jul 06 22:28:51.387932 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.387911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hcq92" event={"ID":"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433","Type":"ContainerStarted","Data":"f7e54db35dd183fdaf506f47a4a038437fc6f76733f18cc75c8bd649d934b501"} Jul 06 22:28:51.973811 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:51.973614 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7985b97cb5-5whbv"] Jul 06 22:28:52.395691 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.395518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94cww" event={"ID":"54117083-55de-423b-8b4b-7c9e01649591","Type":"ContainerStarted","Data":"8ff4224c4ebb07ea02ce939b737270fd96e44b53bd3f783f19d4aa15b5541503"} Jul 06 22:28:52.395691 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.395587 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94cww" event={"ID":"54117083-55de-423b-8b4b-7c9e01649591","Type":"ContainerStarted","Data":"18786ab4d011d54f7a0f596e589dd711292487c57aa9af0ac649d8f5f34f0919"} Jul 06 22:28:52.402354 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.401500 2580 generic.go:356] "Generic (PLEG): container finished" podID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerID="3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e" exitCode=0 Jul 06 22:28:52.402354 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.401608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerDied","Data":"3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e"} Jul 06 22:28:52.405417 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.405390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" event={"ID":"110b2bb5-1ae2-4d9e-8488-0ae89109feae","Type":"ContainerStarted","Data":"d3f6a7dbcad31488631fd3d7b29ff22bcc5c447142803305dacc6ea529dad84f"} Jul 06 22:28:52.408050 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.407800 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" event={"ID":"d95494d6-fb6b-428d-b9cd-a9411a4ec95f","Type":"ContainerStarted","Data":"4343570fa29fc60e936defd8e004a54def8970086fbe7f910fdb9a0e9dfaa653"} Jul 06 22:28:52.412441 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.412382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" event={"ID":"f0155679-45f4-4ca6-addd-f9cf5b85d1bd","Type":"ContainerStarted","Data":"141bd4bb23804773933d29c61ed9d12e6f81db0a37ad06fe97a19b93d8911cee"} Jul 06 22:28:52.412441 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.412436 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" event={"ID":"f0155679-45f4-4ca6-addd-f9cf5b85d1bd","Type":"ContainerStarted","Data":"581e112efe88d25554e702d7d238920e4a77ab7bbb6fcfbe8a231950f40088c8"} Jul 06 22:28:52.412589 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.412463 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" event={"ID":"f0155679-45f4-4ca6-addd-f9cf5b85d1bd","Type":"ContainerStarted","Data":"3cb21ef73a6627f4de578cd5959c46a585d2a0de3b6ee3621f8666de4dba98c1"} Jul 06 22:28:52.416926 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.416563 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-94cww" podStartSLOduration=3.324878642 podStartE2EDuration="4.41654464s" totalImagesPullingTime="987.43744ms" totalInitContainerRuntime="104.228558ms" isStatefulPod=true podCreationTimestamp="2026-07-06 22:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:52.415513965 +0000 UTC m=+68.025515197" watchObservedRunningTime="2026-07-06 22:28:52.41654464 +0000 UTC m=+68.026545851" Jul 06 22:28:52.460028 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:52.459955 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6q2pj" podStartSLOduration=2.569737281 podStartE2EDuration="4.459933462s" totalImagesPullingTime="1.890196181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:52.458206083 +0000 UTC m=+68.068207316" watchObservedRunningTime="2026-07-06 22:28:52.459933462 +0000 UTC m=+68.069934667" Jul 06 22:28:53.140855 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.138859 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t5txl" podStartSLOduration=3.341388472 podStartE2EDuration="5.138833915s" totalImagesPullingTime="1.797445443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:52.480059882 +0000 UTC m=+68.090061091" watchObservedRunningTime="2026-07-06 22:28:53.138833915 +0000 UTC m=+68.748835126" Jul 06 22:28:53.140855 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.139510 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh"] Jul 06 22:28:53.143053 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.142017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.148819 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.147657 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8f7xh\"" Jul 06 22:28:53.148819 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.147823 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e9nlel50pr9e0\"" Jul 06 22:28:53.148819 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.147919 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 22:28:53.148819 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.148044 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 22:28:53.148819 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.148096 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 22:28:53.148819 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.148392 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 22:28:53.151772 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.151719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh"] Jul 06 22:28:53.232435 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.232399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a594377-16e1-4c25-b888-8ff754086a2a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.232622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.232465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3a594377-16e1-4c25-b888-8ff754086a2a-metrics-server-audit-profiles\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.232622 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.232548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3a594377-16e1-4c25-b888-8ff754086a2a-audit-log\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.232734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.232625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-secret-metrics-server-tls\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.232734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.232677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-secret-metrics-server-client-certs\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.232734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.232712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9c28\" (UniqueName: \"kubernetes.io/projected/3a594377-16e1-4c25-b888-8ff754086a2a-kube-api-access-g9c28\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.232876 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.232748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-client-ca-bundle\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.334172 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.334111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a594377-16e1-4c25-b888-8ff754086a2a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.334341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.334205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3a594377-16e1-4c25-b888-8ff754086a2a-metrics-server-audit-profiles\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.334341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.334244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3a594377-16e1-4c25-b888-8ff754086a2a-audit-log\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.334341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.334279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-secret-metrics-server-tls\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.334341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.334326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-secret-metrics-server-client-certs\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.334574 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.334361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9c28\" (UniqueName: \"kubernetes.io/projected/3a594377-16e1-4c25-b888-8ff754086a2a-kube-api-access-g9c28\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.334574 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.334389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-client-ca-bundle\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.335307 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.334907 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a594377-16e1-4c25-b888-8ff754086a2a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.335737 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.335679 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3a594377-16e1-4c25-b888-8ff754086a2a-audit-log\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.336208 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.336184 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3a594377-16e1-4c25-b888-8ff754086a2a-metrics-server-audit-profiles\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.338414 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.338369 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-client-ca-bundle\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.338514 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.338406 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-secret-metrics-server-tls\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.339377 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.339354 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3a594377-16e1-4c25-b888-8ff754086a2a-secret-metrics-server-client-certs\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.343962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.343938 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9c28\" (UniqueName: \"kubernetes.io/projected/3a594377-16e1-4c25-b888-8ff754086a2a-kube-api-access-g9c28\") pod \"metrics-server-7b47fdb9d5-zk8mh\" (UID: \"3a594377-16e1-4c25-b888-8ff754086a2a\") " pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.422601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.421831 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hcq92" event={"ID":"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433","Type":"ContainerStarted","Data":"21e285763b0cf8585b3e87feb5ccaec63054859252da677b71be3215a572c7e6"} Jul 06 22:28:53.422601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.421889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hcq92" event={"ID":"4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433","Type":"ContainerStarted","Data":"94029ca35f67655ddcc2b06960602792ab377b896082124adac53c9a557e1f2a"} Jul 06 22:28:53.442093 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.441441 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hcq92" podStartSLOduration=66.683114644 podStartE2EDuration="1m8.441417401s" totalImagesPullingTime="1.758302757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:53.440815526 +0000 UTC m=+69.050816736" watchObservedRunningTime="2026-07-06 22:28:53.441417401 +0000 UTC m=+69.051418612" Jul 06 22:28:53.456360 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.456101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:28:53.618406 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.618372 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl"] Jul 06 22:28:53.621575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.621549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:28:53.625297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.625274 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jf5l5\"" Jul 06 22:28:53.625412 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.625323 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 22:28:53.637189 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.637148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl"] Jul 06 22:28:53.696315 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.696245 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh"] Jul 06 22:28:53.751180 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.751112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22db243c-957b-406a-b7b8-5da9fa8216d8-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-nw5zl\" (UID: \"22db243c-957b-406a-b7b8-5da9fa8216d8\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:28:53.751344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.751210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5jm8\" (UniqueName: \"kubernetes.io/projected/22db243c-957b-406a-b7b8-5da9fa8216d8-kube-api-access-d5jm8\") pod \"monitoring-plugin-d5dcff7cf-nw5zl\" (UID: \"22db243c-957b-406a-b7b8-5da9fa8216d8\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:28:53.793439 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.793368 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-vmt4j"] Jul 06 22:28:53.797311 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.796312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-vmt4j" Jul 06 22:28:53.800503 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.799071 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2l2p5\"" Jul 06 22:28:53.800503 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.799368 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 22:28:53.800503 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.799403 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 22:28:53.806078 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.806016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-vmt4j"] Jul 06 22:28:53.853422 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.852535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22db243c-957b-406a-b7b8-5da9fa8216d8-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-nw5zl\" (UID: \"22db243c-957b-406a-b7b8-5da9fa8216d8\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:28:53.853422 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.852592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dclz7\" (UniqueName: \"kubernetes.io/projected/d8f92260-54c5-48bf-aebd-5960e0f85343-kube-api-access-dclz7\") pod \"downloads-685bfb5f64-vmt4j\" (UID: \"d8f92260-54c5-48bf-aebd-5960e0f85343\") " pod="openshift-console/downloads-685bfb5f64-vmt4j" Jul 06 22:28:53.853422 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.852648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5jm8\" (UniqueName: \"kubernetes.io/projected/22db243c-957b-406a-b7b8-5da9fa8216d8-kube-api-access-d5jm8\") pod \"monitoring-plugin-d5dcff7cf-nw5zl\" (UID: \"22db243c-957b-406a-b7b8-5da9fa8216d8\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:28:53.856511 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.856441 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22db243c-957b-406a-b7b8-5da9fa8216d8-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-nw5zl\" (UID: \"22db243c-957b-406a-b7b8-5da9fa8216d8\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:28:53.868413 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.868378 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5jm8\" (UniqueName: \"kubernetes.io/projected/22db243c-957b-406a-b7b8-5da9fa8216d8-kube-api-access-d5jm8\") pod \"monitoring-plugin-d5dcff7cf-nw5zl\" (UID: \"22db243c-957b-406a-b7b8-5da9fa8216d8\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:28:53.936154 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.936113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:28:53.954010 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.953944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dclz7\" (UniqueName: \"kubernetes.io/projected/d8f92260-54c5-48bf-aebd-5960e0f85343-kube-api-access-dclz7\") pod \"downloads-685bfb5f64-vmt4j\" (UID: \"d8f92260-54c5-48bf-aebd-5960e0f85343\") " pod="openshift-console/downloads-685bfb5f64-vmt4j" Jul 06 22:28:53.972677 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:53.972648 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dclz7\" (UniqueName: \"kubernetes.io/projected/d8f92260-54c5-48bf-aebd-5960e0f85343-kube-api-access-dclz7\") pod \"downloads-685bfb5f64-vmt4j\" (UID: \"d8f92260-54c5-48bf-aebd-5960e0f85343\") " pod="openshift-console/downloads-685bfb5f64-vmt4j" Jul 06 22:28:54.078891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.078712 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75b787f7bb-kqxsj"] Jul 06 22:28:54.083542 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.082932 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.086246 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.086222 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 22:28:54.086705 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.086684 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 22:28:54.087009 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.086987 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 22:28:54.087512 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.087478 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 22:28:54.087644 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.087624 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 22:28:54.087702 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.087657 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-x65d7\"" Jul 06 22:28:54.093638 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.093610 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 22:28:54.093839 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.093808 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75b787f7bb-kqxsj"] Jul 06 22:28:54.114532 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.114495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-vmt4j" Jul 06 22:28:54.155897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.155505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.155897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.155561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkp6\" (UniqueName: \"kubernetes.io/projected/b40e2735-9974-4e6b-b6ab-5fab1367703b-kube-api-access-fqkp6\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.155897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.155597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-serving-certs-ca-bundle\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.155897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.155622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-federate-client-tls\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.155897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.155671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.155897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.155738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-metrics-client-ca\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.155897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.155763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-secret-telemeter-client\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.155897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.155794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-telemeter-client-tls\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.256680 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.256594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-serving-certs-ca-bundle\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.256680 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.256648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-federate-client-tls\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.256888 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.256685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.256888 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.256737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-metrics-client-ca\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.256888 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.256768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-secret-telemeter-client\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.256888 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.256800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-telemeter-client-tls\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.256888 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.256879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.257157 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.256918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkp6\" (UniqueName: \"kubernetes.io/projected/b40e2735-9974-4e6b-b6ab-5fab1367703b-kube-api-access-fqkp6\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.257996 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.257515 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-serving-certs-ca-bundle\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.257996 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.257687 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-metrics-client-ca\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.258626 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.258584 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40e2735-9974-4e6b-b6ab-5fab1367703b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.260671 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.260510 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-secret-telemeter-client\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.260671 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.260542 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-federate-client-tls\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.261004 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.260950 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.261004 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.260954 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b40e2735-9974-4e6b-b6ab-5fab1367703b-telemeter-client-tls\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.265710 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.265662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkp6\" (UniqueName: \"kubernetes.io/projected/b40e2735-9974-4e6b-b6ab-5fab1367703b-kube-api-access-fqkp6\") pod \"telemeter-client-75b787f7bb-kqxsj\" (UID: \"b40e2735-9974-4e6b-b6ab-5fab1367703b\") " pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.399015 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.398975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" Jul 06 22:28:54.427573 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:54.427536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" event={"ID":"3a594377-16e1-4c25-b888-8ff754086a2a","Type":"ContainerStarted","Data":"c6112a407bb3917141e1d2781a6ef8379b64467b4811036bc2571c53fd4d29e1"} Jul 06 22:28:55.116910 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.116553 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:28:55.121779 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.121042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.124968 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.124946 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 22:28:55.125198 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.125157 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 22:28:55.125356 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.124946 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 22:28:55.125390 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.125373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 22:28:55.126891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126452 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-twx67\"" Jul 06 22:28:55.126891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126603 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 22:28:55.126891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126603 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 22:28:55.126891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126704 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 22:28:55.126891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126850 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 22:28:55.126891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126858 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 22:28:55.126891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126886 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1mvt7lfvfme4k\"" Jul 06 22:28:55.127322 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126985 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 22:28:55.127322 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.126999 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 22:28:55.127322 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.127244 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 22:28:55.144982 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.144943 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:28:55.145199 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.145178 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 22:28:55.145462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.145442 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 22:28:55.169176 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169613 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169613 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169716 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169889 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169889 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169889 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169889 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.169889 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170122 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170122 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170122 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.169978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170122 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.170003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170122 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.170042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhzw2\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-kube-api-access-xhzw2\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170122 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.170074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170122 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.170116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170468 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.170143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.170468 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.170205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271182 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhzw2\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-kube-api-access-xhzw2\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271603 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271603 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271603 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271603 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271603 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.271897 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.273008 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.271902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.273008 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.272085 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.273008 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.272459 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.274202 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.274177 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.274633 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.274604 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.276293 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.275690 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.276293 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.275906 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.276293 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.276231 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.276293 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.276239 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.276293 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.276259 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.277462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.276643 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.277462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.276875 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.277462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.276955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.277462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.277251 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.277462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.277388 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.277462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.277446 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.277832 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.277699 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.278339 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.278319 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.278828 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.278783 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.280038 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.280013 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhzw2\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-kube-api-access-xhzw2\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.280148 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.280065 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:55.440539 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:55.440504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:28:56.358010 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:56.357980 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8qbqw" Jul 06 22:28:58.118646 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.118598 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl"] Jul 06 22:28:58.121299 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:58.121238 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22db243c_957b_406a_b7b8_5da9fa8216d8.slice/crio-32dbedccf03e5d3953b1c5d72b08a7c835b87859492a9082bfacc5c67083dc4d WatchSource:0}: Error finding container 32dbedccf03e5d3953b1c5d72b08a7c835b87859492a9082bfacc5c67083dc4d: Status 404 returned error can't find the container with id 32dbedccf03e5d3953b1c5d72b08a7c835b87859492a9082bfacc5c67083dc4d Jul 06 22:28:58.368143 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.368112 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75b787f7bb-kqxsj"] Jul 06 22:28:58.368752 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:28:58.368710 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40e2735_9974_4e6b_b6ab_5fab1367703b.slice/crio-66ea2c515b5e536b0b0b5204ceb5da2fa06f193870ab914562c81e93386ac64b WatchSource:0}: Error finding container 66ea2c515b5e536b0b0b5204ceb5da2fa06f193870ab914562c81e93386ac64b: Status 404 returned error can't find the container with id 66ea2c515b5e536b0b0b5204ceb5da2fa06f193870ab914562c81e93386ac64b Jul 06 22:28:58.369534 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.369240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-vmt4j"] Jul 06 22:28:58.371801 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.371446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:28:58.448916 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.448850 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4qqdf" event={"ID":"f20455a2-a971-4b8f-bfab-879716005fbf","Type":"ContainerStarted","Data":"dc01c4cb4ee9d6b8d2ddfe86228cc4299ff048c894cbb02fa3815579889cd972"} Jul 06 22:28:58.451862 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.451833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" event={"ID":"d95494d6-fb6b-428d-b9cd-a9411a4ec95f","Type":"ContainerStarted","Data":"8050ba8161fe8fbe39adfa7f8805be078cd398a7c579c788a94b65de558319dd"} Jul 06 22:28:58.451961 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.451907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" event={"ID":"d95494d6-fb6b-428d-b9cd-a9411a4ec95f","Type":"ContainerStarted","Data":"cc094ef6499783e948391a110ed605c64fa116dd5a87382f01ca07328c6ef4eb"} Jul 06 22:28:58.451961 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.451948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" event={"ID":"d95494d6-fb6b-428d-b9cd-a9411a4ec95f","Type":"ContainerStarted","Data":"0aa4f188c3fb6a2f63b37ac36551e0a468152900c98f506178a43b946fc9979e"} Jul 06 22:28:58.454821 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.454799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8wq78" event={"ID":"c9a54d7a-954e-405c-a0c5-8fb9a3d37077","Type":"ContainerStarted","Data":"3911b709ea2f7713f94d71acadeeae111e279b2cf5b9cea281c1d445bcd314c6"} Jul 06 22:28:58.455360 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.455325 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:28:58.456706 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.456685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" event={"ID":"b40e2735-9974-4e6b-b6ab-5fab1367703b","Type":"ContainerStarted","Data":"66ea2c515b5e536b0b0b5204ceb5da2fa06f193870ab914562c81e93386ac64b"} Jul 06 22:28:58.458955 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.458459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerStarted","Data":"aca56a2581026888faf8d21b38b84f5fd00ad77a65ddbbdfaabcfc411a99018d"} Jul 06 22:28:58.461400 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.461377 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" event={"ID":"22db243c-957b-406a-b7b8-5da9fa8216d8","Type":"ContainerStarted","Data":"32dbedccf03e5d3953b1c5d72b08a7c835b87859492a9082bfacc5c67083dc4d"} Jul 06 22:28:58.462743 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.462712 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-vmt4j" event={"ID":"d8f92260-54c5-48bf-aebd-5960e0f85343","Type":"ContainerStarted","Data":"3bd7c580a17d645f77328475711997018acd179ac4820cc5ae1956057a939289"} Jul 06 22:28:58.466371 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.466348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerStarted","Data":"953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a"} Jul 06 22:28:58.466472 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.466414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerStarted","Data":"54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29"} Jul 06 22:28:58.466472 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.466458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerStarted","Data":"ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6"} Jul 06 22:28:58.466567 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.466496 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerStarted","Data":"cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e"} Jul 06 22:28:58.466567 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.466535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerStarted","Data":"3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51"} Jul 06 22:28:58.469125 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.469102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" event={"ID":"3a594377-16e1-4c25-b888-8ff754086a2a","Type":"ContainerStarted","Data":"134199d0d1001bf16f5aacbf008d85f3c986f1f96c065bd353e1fc2e414f32ca"} Jul 06 22:28:58.481384 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.481333 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4qqdf" podStartSLOduration=66.53177348 podStartE2EDuration="1m13.481315302s" totalImagesPullingTime="6.949541822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:58.481266636 +0000 UTC m=+74.091267845" watchObservedRunningTime="2026-07-06 22:28:58.481315302 +0000 UTC m=+74.091316514" Jul 06 22:28:58.518543 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.518487 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" podStartSLOduration=1.21461812 podStartE2EDuration="5.518472198s" totalImagesPullingTime="4.303854078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:58.516206044 +0000 UTC m=+74.126207257" watchObservedRunningTime="2026-07-06 22:28:58.518472198 +0000 UTC m=+74.128473389" Jul 06 22:28:58.585113 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:58.585059 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8wq78" podStartSLOduration=66.638798433 podStartE2EDuration="1m13.585039735s" totalImagesPullingTime="6.946241302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:58.583971807 +0000 UTC m=+74.193973016" watchObservedRunningTime="2026-07-06 22:28:58.585039735 +0000 UTC m=+74.195040944" Jul 06 22:28:59.477192 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:59.477055 2580 generic.go:356] "Generic (PLEG): container finished" podID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" exitCode=0 Jul 06 22:28:59.478117 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:28:59.477671 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerDied","Data":"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a"} Jul 06 22:29:00.484126 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.484060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" event={"ID":"22db243c-957b-406a-b7b8-5da9fa8216d8","Type":"ContainerStarted","Data":"2205ecaba8b62d3a90c9714c8a4ffb6a28e355de3d2dbb365d076f523554dc85"} Jul 06 22:29:00.484618 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.484155 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:29:00.489105 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.489017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerStarted","Data":"ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9"} Jul 06 22:29:00.489953 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.489925 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" Jul 06 22:29:00.492734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.492712 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" event={"ID":"d95494d6-fb6b-428d-b9cd-a9411a4ec95f","Type":"ContainerStarted","Data":"b7d7d3180ce266a7498ed1d4c9db1b0181f782194d69659e53caea20155b6c44"} Jul 06 22:29:00.492863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.492775 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" event={"ID":"d95494d6-fb6b-428d-b9cd-a9411a4ec95f","Type":"ContainerStarted","Data":"1146b4fdbf99f6ceeb4563dc9c5131c1a2930af8011303c50bf6680faf920e71"} Jul 06 22:29:00.492863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.492815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" event={"ID":"d95494d6-fb6b-428d-b9cd-a9411a4ec95f","Type":"ContainerStarted","Data":"4d26b6decb4719efcbb7dd43fb521356ecd044e6897a6a90cb7607d0a632bfa8"} Jul 06 22:29:00.504427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.504378 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nw5zl" podStartSLOduration=5.972509464 podStartE2EDuration="7.504358462s" totalImagesPullingTime="1.531848998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:00.501837161 +0000 UTC m=+76.111838371" watchObservedRunningTime="2026-07-06 22:29:00.504358462 +0000 UTC m=+76.114359672" Jul 06 22:29:00.543599 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.543551 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" podStartSLOduration=3.269010178 podStartE2EDuration="10.543532494s" totalImagesPullingTime="7.274522316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:00.54297633 +0000 UTC m=+76.152977567" watchObservedRunningTime="2026-07-06 22:29:00.543532494 +0000 UTC m=+76.153533702" Jul 06 22:29:00.576588 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:00.576527 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.4103666329999998 podStartE2EDuration="11.576503008s" totalImagesPullingTime="8.112445523s" totalInitContainerRuntime="53.690852ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:00.573569993 +0000 UTC m=+76.183571235" watchObservedRunningTime="2026-07-06 22:29:00.576503008 +0000 UTC m=+76.186504217" Jul 06 22:29:01.501938 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:01.501630 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" event={"ID":"b40e2735-9974-4e6b-b6ab-5fab1367703b","Type":"ContainerStarted","Data":"e8932c14fd71e3ab7b1faf92e13f4d9c7269458cfdf8943beb1ba93ede2e67a8"} Jul 06 22:29:01.501938 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:01.501721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" event={"ID":"b40e2735-9974-4e6b-b6ab-5fab1367703b","Type":"ContainerStarted","Data":"dcb3fc4991b54e6566382dc086443739489d6f63be812b47b876459a53dfe5e6"} Jul 06 22:29:01.501938 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:01.501755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" event={"ID":"b40e2735-9974-4e6b-b6ab-5fab1367703b","Type":"ContainerStarted","Data":"1158a83f65684008cdaac62cf8ce8c85f5546e03bc7902332fcfaa31fcf1cedb"} Jul 06 22:29:01.501938 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:01.501925 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:29:01.531184 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:01.531080 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75b787f7bb-kqxsj" podStartSLOduration=5.383191427 podStartE2EDuration="7.53105726s" totalImagesPullingTime="2.147865833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:01.52922017 +0000 UTC m=+77.139221399" watchObservedRunningTime="2026-07-06 22:29:01.53105726 +0000 UTC m=+77.141058470" Jul 06 22:29:04.525369 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:04.525298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerStarted","Data":"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3"} Jul 06 22:29:04.525822 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:04.525425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerStarted","Data":"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336"} Jul 06 22:29:04.525822 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:04.525504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerStarted","Data":"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc"} Jul 06 22:29:04.525822 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:04.525545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerStarted","Data":"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20"} Jul 06 22:29:04.525822 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:04.525583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerStarted","Data":"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59"} Jul 06 22:29:04.525822 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:04.525629 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerStarted","Data":"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f"} Jul 06 22:29:04.557556 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:04.557501 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.2559209639999995 podStartE2EDuration="9.557484193s" totalImagesPullingTime="4.28128012s" totalInitContainerRuntime="20.283109ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:04.554396645 +0000 UTC m=+80.164397855" watchObservedRunningTime="2026-07-06 22:29:04.557484193 +0000 UTC m=+80.167485402" Jul 06 22:29:05.441088 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:05.441047 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:29:07.516320 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:07.516220 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7985b97cb5-5whbv" Jul 06 22:29:13.456342 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:13.456309 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:29:13.456941 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:13.456387 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:29:16.589870 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:16.589827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-vmt4j" event={"ID":"d8f92260-54c5-48bf-aebd-5960e0f85343","Type":"ContainerStarted","Data":"b8e509e9f844e858113ccd386f7766f367fd3af4a81b4cad1a2afa710ddbe5f3"} Jul 06 22:29:16.590476 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:16.590449 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-vmt4j" Jul 06 22:29:16.599700 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:16.599649 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-vmt4j" Jul 06 22:29:16.610045 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:16.609994 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-vmt4j" podStartSLOduration=5.856218499 podStartE2EDuration="23.609974017s" totalImagesPullingTime="17.753755518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:16.60949329 +0000 UTC m=+92.219494514" watchObservedRunningTime="2026-07-06 22:29:16.609974017 +0000 UTC m=+92.219975227" Jul 06 22:29:30.495951 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:30.495921 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8wq78" Jul 06 22:29:33.463072 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:33.463045 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:29:33.467283 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:33.467259 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b47fdb9d5-zk8mh" Jul 06 22:29:55.441060 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:55.441028 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:29:55.493764 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:55.493740 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:29:55.780143 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:29:55.780060 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:09.194226 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.194191 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:30:09.194796 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.194529 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="alertmanager" containerID="cri-o://3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51" gracePeriod=120 Jul 06 22:30:09.194796 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.194564 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy-metric" containerID="cri-o://953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a" gracePeriod=120 Jul 06 22:30:09.194796 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.194597 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="prom-label-proxy" containerID="cri-o://ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9" gracePeriod=120 Jul 06 22:30:09.194796 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.194578 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="config-reloader" containerID="cri-o://cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e" gracePeriod=120 Jul 06 22:30:09.194796 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.194576 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy-web" containerID="cri-o://ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6" gracePeriod=120 Jul 06 22:30:09.194796 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.194634 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy" containerID="cri-o://54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29" gracePeriod=120 Jul 06 22:30:09.822581 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822550 2580 generic.go:356] "Generic (PLEG): container finished" podID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerID="ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9" exitCode=0 Jul 06 22:30:09.822581 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822574 2580 generic.go:356] "Generic (PLEG): container finished" podID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerID="953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a" exitCode=0 Jul 06 22:30:09.822581 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822583 2580 generic.go:356] "Generic (PLEG): container finished" podID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerID="54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29" exitCode=0 Jul 06 22:30:09.822581 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822590 2580 generic.go:356] "Generic (PLEG): container finished" podID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerID="cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e" exitCode=0 Jul 06 22:30:09.822863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822596 2580 generic.go:356] "Generic (PLEG): container finished" podID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerID="3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51" exitCode=0 Jul 06 22:30:09.822863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerDied","Data":"ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9"} Jul 06 22:30:09.822863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822682 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerDied","Data":"953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a"} Jul 06 22:30:09.822863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerDied","Data":"54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29"} Jul 06 22:30:09.822863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerDied","Data":"cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e"} Jul 06 22:30:09.822863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:09.822759 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerDied","Data":"3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51"} Jul 06 22:30:10.440438 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.440413 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.467343 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467313 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-trusted-ca-bundle\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467511 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467390 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-cluster-tls-config\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467511 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467457 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-web-config\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467511 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467488 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-out\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467701 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467580 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-volume\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467701 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467657 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-metrics-client-ca\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467701 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467696 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-tls-assets\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467853 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467724 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-main-db\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467853 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467843 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-main-tls\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467954 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467879 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467954 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467909 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-web\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.467954 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467944 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97c5n\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-kube-api-access-97c5n\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.468095 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467985 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy\") pod \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\" (UID: \"f46549a4-1a24-4c36-8eb6-e93a26e8ec44\") " Jul 06 22:30:10.470846 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.467769 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:10.470846 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.468355 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.470846 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.469291 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:10.472448 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.471990 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:30:10.472448 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.472407 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:30:10.473182 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.473142 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-out" (OuterVolumeSpecName: "config-out") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:30:10.473527 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.473493 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-kube-api-access-97c5n" (OuterVolumeSpecName: "kube-api-access-97c5n") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "kube-api-access-97c5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:30:10.474070 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.473915 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:10.475031 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.474993 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:10.475610 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.475555 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:10.475931 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.475882 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-volume" (OuterVolumeSpecName: "config-volume") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:10.476033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.475231 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:10.479971 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.479945 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:10.490290 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.490265 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-web-config" (OuterVolumeSpecName: "web-config") pod "f46549a4-1a24-4c36-8eb6-e93a26e8ec44" (UID: "f46549a4-1a24-4c36-8eb6-e93a26e8ec44"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568776 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-web-config\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568806 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-out\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568815 2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-config-volume\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568828 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-metrics-client-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568839 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-tls-assets\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568847 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-alertmanager-main-db\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568856 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-main-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568867 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568875 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.568877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568885 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97c5n\" (UniqueName: \"kubernetes.io/projected/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-kube-api-access-97c5n\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.569509 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568896 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.569509 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.568908 2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f46549a4-1a24-4c36-8eb6-e93a26e8ec44-cluster-tls-config\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:10.829986 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.829899 2580 generic.go:356] "Generic (PLEG): container finished" podID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerID="ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6" exitCode=0 Jul 06 22:30:10.829986 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.829966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerDied","Data":"ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6"} Jul 06 22:30:10.830197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.829986 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.830197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.830055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f46549a4-1a24-4c36-8eb6-e93a26e8ec44","Type":"ContainerDied","Data":"060208148e7b3984e47c1b15cf933eb6f52e350251c38d261d6e61da955c595e"} Jul 06 22:30:10.830197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.830090 2580 scope.go:117] "RemoveContainer" containerID="ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9" Jul 06 22:30:10.838797 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.838779 2580 scope.go:117] "RemoveContainer" containerID="953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a" Jul 06 22:30:10.845958 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.845937 2580 scope.go:117] "RemoveContainer" containerID="54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29" Jul 06 22:30:10.853261 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.853215 2580 scope.go:117] "RemoveContainer" containerID="ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6" Jul 06 22:30:10.854623 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.854572 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:30:10.857760 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.857739 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:30:10.861159 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.861143 2580 scope.go:117] "RemoveContainer" containerID="cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e" Jul 06 22:30:10.867677 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.867662 2580 scope.go:117] "RemoveContainer" containerID="3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51" Jul 06 22:30:10.874504 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.874473 2580 scope.go:117] "RemoveContainer" containerID="3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e" Jul 06 22:30:10.881104 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.881089 2580 scope.go:117] "RemoveContainer" containerID="ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9" Jul 06 22:30:10.881444 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:10.881409 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9\": container with ID starting with ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9 not found: ID does not exist" containerID="ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9" Jul 06 22:30:10.881572 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.881447 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9"} err="failed to get container status \"ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9\": rpc error: code = NotFound desc = could not find container \"ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9\": container with ID starting with ac877c7656c00e45a1d6ab7a133cfe679fe1852559f172d08de2aef0819528f9 not found: ID does not exist" Jul 06 22:30:10.881572 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.881492 2580 scope.go:117] "RemoveContainer" containerID="953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a" Jul 06 22:30:10.882351 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:10.882323 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a\": container with ID starting with 953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a not found: ID does not exist" containerID="953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a" Jul 06 22:30:10.882426 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.882360 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a"} err="failed to get container status \"953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a\": rpc error: code = NotFound desc = could not find container \"953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a\": container with ID starting with 953cea36d1173a0eaec9704262c34069e660cdf33c2545d06078b66d8970d90a not found: ID does not exist" Jul 06 22:30:10.882426 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.882383 2580 scope.go:117] "RemoveContainer" containerID="54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29" Jul 06 22:30:10.882716 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:10.882698 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29\": container with ID starting with 54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29 not found: ID does not exist" containerID="54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29" Jul 06 22:30:10.882779 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.882724 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29"} err="failed to get container status \"54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29\": rpc error: code = NotFound desc = could not find container \"54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29\": container with ID starting with 54da271ed5686d683b4d7c37c4d6c09940f5e0c5c692d09dbab901db2b5a3b29 not found: ID does not exist" Jul 06 22:30:10.882779 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.882742 2580 scope.go:117] "RemoveContainer" containerID="ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6" Jul 06 22:30:10.883009 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:10.882987 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6\": container with ID starting with ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6 not found: ID does not exist" containerID="ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6" Jul 06 22:30:10.883056 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.883018 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6"} err="failed to get container status \"ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6\": rpc error: code = NotFound desc = could not find container \"ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6\": container with ID starting with ab9c4506dd85990e7710476c946259a58e9c72c3723535032eb9c4c9cd9c21e6 not found: ID does not exist" Jul 06 22:30:10.883056 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.883039 2580 scope.go:117] "RemoveContainer" containerID="cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e" Jul 06 22:30:10.883309 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:10.883293 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e\": container with ID starting with cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e not found: ID does not exist" containerID="cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e" Jul 06 22:30:10.883366 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.883312 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e"} err="failed to get container status \"cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e\": rpc error: code = NotFound desc = could not find container \"cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e\": container with ID starting with cbe58763ffb8a9d4ecf87e7d8c30cd38ef5fbda43db0918e14f62a2b1b363d0e not found: ID does not exist" Jul 06 22:30:10.883366 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.883327 2580 scope.go:117] "RemoveContainer" containerID="3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51" Jul 06 22:30:10.883552 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:10.883531 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51\": container with ID starting with 3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51 not found: ID does not exist" containerID="3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51" Jul 06 22:30:10.883590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.883559 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51"} err="failed to get container status \"3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51\": rpc error: code = NotFound desc = could not find container \"3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51\": container with ID starting with 3834d0edafb6f7cf488c5b2335b46fed6d681ff435f1d205e3d58db67400ef51 not found: ID does not exist" Jul 06 22:30:10.883590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.883579 2580 scope.go:117] "RemoveContainer" containerID="3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e" Jul 06 22:30:10.883664 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.883653 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.883984 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884003 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:10.884012 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e\": container with ID starting with 3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e not found: ID does not exist" containerID="3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884025 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="init-config-reloader" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884038 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="init-config-reloader" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884037 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e"} err="failed to get container status \"3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e\": rpc error: code = NotFound desc = could not find container \"3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e\": container with ID starting with 3c552f10b8e74507253754882d2767653959cb0211d49942b07a86478dead96e not found: ID does not exist" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884049 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy-metric" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884059 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy-metric" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884079 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy-web" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884088 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy-web" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884105 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="config-reloader" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884149 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="config-reloader" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884232 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="prom-label-proxy" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884259 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="prom-label-proxy" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884297 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="alertmanager" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884328 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="alertmanager" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884412 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy-metric" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884427 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="config-reloader" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884437 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy-web" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884446 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="prom-label-proxy" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884455 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="alertmanager" Jul 06 22:30:10.885488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.884464 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" containerName="kube-rbac-proxy" Jul 06 22:30:10.888611 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.888594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.890996 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.890976 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 22:30:10.891071 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.891011 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 22:30:10.891071 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.891055 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 22:30:10.891189 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.891098 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 22:30:10.891386 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.891359 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 22:30:10.891467 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.891452 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 22:30:10.891524 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.891510 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-22pmh\"" Jul 06 22:30:10.891631 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.891614 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 22:30:10.892190 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.892155 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 22:30:10.897958 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.897940 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 22:30:10.904437 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.904417 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:30:10.971877 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.971843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.971885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.971908 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.971975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-config-volume\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972217 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972217 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972217 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-web-config\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972217 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972353 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pch8j\" (UniqueName: \"kubernetes.io/projected/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-kube-api-access-pch8j\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972353 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-config-out\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:10.972353 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:10.972280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.002606 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.002575 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f46549a4-1a24-4c36-8eb6-e93a26e8ec44" path="/var/lib/kubelet/pods/f46549a4-1a24-4c36-8eb6-e93a26e8ec44/volumes" Jul 06 22:30:11.073639 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.073604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-web-config\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.073639 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.073642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.073823 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.073660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pch8j\" (UniqueName: \"kubernetes.io/projected/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-kube-api-access-pch8j\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.073823 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.073687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-config-out\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.073823 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.073807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.073960 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.073886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.073960 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.073946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.074081 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.073985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.074081 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.074015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-config-volume\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.074081 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.074057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.074262 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.074085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.074262 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.074121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.074262 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.074148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.074262 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.074159 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.074522 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.074495 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.076475 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.076444 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-config-out\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.076887 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.076805 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.076887 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.076815 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.077061 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.077003 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.077118 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.077075 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.077118 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.077091 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-web-config\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.077218 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.077117 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.077316 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.077297 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.077400 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.077384 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.078782 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.078759 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-config-volume\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.082671 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.082627 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pch8j\" (UniqueName: \"kubernetes.io/projected/57995eeb-c068-4e4e-aa55-0e56ab6cbbdb-kube-api-access-pch8j\") pod \"alertmanager-main-0\" (UID: \"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.198145 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.198114 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:30:11.337261 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.337181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:30:11.340316 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:30:11.340288 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57995eeb_c068_4e4e_aa55_0e56ab6cbbdb.slice/crio-a879b8cb9efa910430bd7ce44566ad9f1779942e8349b39d97b9a9df2f979481 WatchSource:0}: Error finding container a879b8cb9efa910430bd7ce44566ad9f1779942e8349b39d97b9a9df2f979481: Status 404 returned error can't find the container with id a879b8cb9efa910430bd7ce44566ad9f1779942e8349b39d97b9a9df2f979481 Jul 06 22:30:11.834931 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.834899 2580 generic.go:356] "Generic (PLEG): container finished" podID="57995eeb-c068-4e4e-aa55-0e56ab6cbbdb" containerID="010b6340d6d54afcc353a5799022c222e6ef02398daba5d8a0fa271d2b309713" exitCode=0 Jul 06 22:30:11.835390 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.834984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb","Type":"ContainerDied","Data":"010b6340d6d54afcc353a5799022c222e6ef02398daba5d8a0fa271d2b309713"} Jul 06 22:30:11.835390 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:11.835052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb","Type":"ContainerStarted","Data":"a879b8cb9efa910430bd7ce44566ad9f1779942e8349b39d97b9a9df2f979481"} Jul 06 22:30:12.842910 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:12.842875 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb","Type":"ContainerStarted","Data":"7384d5f5d252c100e745b67c1a83e0f7fbdb17d5504793087f2aeed6bf7ae279"} Jul 06 22:30:12.843297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:12.842968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb","Type":"ContainerStarted","Data":"b6768bb46588f358c43ffb54c04132b5d3e25afff39447197b53b9141b930896"} Jul 06 22:30:12.843297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:12.842995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb","Type":"ContainerStarted","Data":"78b7bf9fb36f8e98153d9d0a0ad708d43f9f3d573f5e86c070281a54edc9efa8"} Jul 06 22:30:12.843297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:12.843021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb","Type":"ContainerStarted","Data":"d31fbd2092c8fdb64f8c3c6a84625546133b7ad140871730e81f47ee05fa4abc"} Jul 06 22:30:12.843297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:12.843047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb","Type":"ContainerStarted","Data":"1637279c214875a1d324f36f2cf7c563da4d34ffbf590b97210e8c91bf1ec800"} Jul 06 22:30:12.843297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:12.843069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57995eeb-c068-4e4e-aa55-0e56ab6cbbdb","Type":"ContainerStarted","Data":"3c82eb28d82b731777223f778ed4a771eeff6eedb63f75936782d0de3aeb7561"} Jul 06 22:30:12.873081 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:12.873024 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.861585702 podStartE2EDuration="2.873003345s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.417643ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:12.870805506 +0000 UTC m=+148.480806737" watchObservedRunningTime="2026-07-06 22:30:12.873003345 +0000 UTC m=+148.483004554" Jul 06 22:30:13.431945 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.431911 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:30:13.432596 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.432541 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="prometheus" containerID="cri-o://cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" gracePeriod=600 Jul 06 22:30:13.432996 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.432944 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="thanos-sidecar" containerID="cri-o://58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" gracePeriod=600 Jul 06 22:30:13.433101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.432973 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="config-reloader" containerID="cri-o://0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" gracePeriod=600 Jul 06 22:30:13.433101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.432944 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy" containerID="cri-o://3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" gracePeriod=600 Jul 06 22:30:13.433258 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.432925 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy-web" containerID="cri-o://0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" gracePeriod=600 Jul 06 22:30:13.433258 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.432732 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" gracePeriod=600 Jul 06 22:30:13.686444 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.686383 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:13.800544 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.800514 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-0\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.800544 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.800545 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-kube-rbac-proxy\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.800775 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.800569 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-trusted-ca-bundle\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.800775 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.800587 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-serving-certs-ca-bundle\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.800775 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.800609 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-web-config\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.800775 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.800632 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.800775 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.800655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-thanos-prometheus-http-client-file\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.801071 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801000 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:13.801124 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801080 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-grpc-tls\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.801197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801125 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-db\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.801197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801157 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-2\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.801301 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801212 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-tls-assets\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.801301 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801238 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:13.801301 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801250 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-metrics-client-certs\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801527 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801590 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801633 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-tls\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801679 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801704 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-metrics-client-ca\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801738 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-kubelet-serving-ca-bundle\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801764 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhzw2\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-kube-api-access-xhzw2\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801822 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-1\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.801876 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config-out\") pod \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\" (UID: \"af7ee67a-77a4-42e4-a6f3-c89df31a13f3\") " Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.802158 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.802200 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.802217 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.802353 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.802373 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.802981 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:13.803387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.803390 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:13.804292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.803770 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:30:13.804292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.803987 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.804431 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.804406 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.804614 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.804562 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.804706 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.804665 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config" (OuterVolumeSpecName: "config") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.804843 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.804819 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.805371 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.805344 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:30:13.805683 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.805657 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-kube-api-access-xhzw2" (OuterVolumeSpecName: "kube-api-access-xhzw2") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "kube-api-access-xhzw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:30:13.805966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.805944 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.806432 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.806402 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config-out" (OuterVolumeSpecName: "config-out") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:30:13.806526 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.806404 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.806526 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.806506 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.815195 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.815158 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-web-config" (OuterVolumeSpecName: "web-config") pod "af7ee67a-77a4-42e4-a6f3-c89df31a13f3" (UID: "af7ee67a-77a4-42e4-a6f3-c89df31a13f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:30:13.850863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850832 2580 generic.go:356] "Generic (PLEG): container finished" podID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" exitCode=0 Jul 06 22:30:13.850863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850857 2580 generic.go:356] "Generic (PLEG): container finished" podID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" exitCode=0 Jul 06 22:30:13.850863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850864 2580 generic.go:356] "Generic (PLEG): container finished" podID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" exitCode=0 Jul 06 22:30:13.850863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850869 2580 generic.go:356] "Generic (PLEG): container finished" podID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" exitCode=0 Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850875 2580 generic.go:356] "Generic (PLEG): container finished" podID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" exitCode=0 Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850880 2580 generic.go:356] "Generic (PLEG): container finished" podID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" exitCode=0 Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerDied","Data":"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3"} Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850918 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.850993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerDied","Data":"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336"} Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.851039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerDied","Data":"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc"} Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.851053 2580 scope.go:117] "RemoveContainer" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.851081 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerDied","Data":"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20"} Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.851123 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerDied","Data":"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59"} Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.851181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerDied","Data":"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f"} Jul 06 22:30:13.851433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.851222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af7ee67a-77a4-42e4-a6f3-c89df31a13f3","Type":"ContainerDied","Data":"aca56a2581026888faf8d21b38b84f5fd00ad77a65ddbbdfaabcfc411a99018d"} Jul 06 22:30:13.859255 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.859230 2580 scope.go:117] "RemoveContainer" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" Jul 06 22:30:13.866551 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.866536 2580 scope.go:117] "RemoveContainer" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" Jul 06 22:30:13.873443 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.873344 2580 scope.go:117] "RemoveContainer" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" Jul 06 22:30:13.876538 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.876519 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:30:13.880026 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.880007 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:30:13.881522 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.881511 2580 scope.go:117] "RemoveContainer" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" Jul 06 22:30:13.888065 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.888051 2580 scope.go:117] "RemoveContainer" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" Jul 06 22:30:13.895418 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.895396 2580 scope.go:117] "RemoveContainer" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" Jul 06 22:30:13.902144 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.902125 2580 scope.go:117] "RemoveContainer" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" Jul 06 22:30:13.902462 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:13.902437 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": container with ID starting with 282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3 not found: ID does not exist" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" Jul 06 22:30:13.902509 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.902474 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3"} err="failed to get container status \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": rpc error: code = NotFound desc = could not find container \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": container with ID starting with 282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3 not found: ID does not exist" Jul 06 22:30:13.902509 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.902499 2580 scope.go:117] "RemoveContainer" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" Jul 06 22:30:13.902766 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:13.902740 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": container with ID starting with 3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336 not found: ID does not exist" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" Jul 06 22:30:13.902866 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.902768 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336"} err="failed to get container status \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": rpc error: code = NotFound desc = could not find container \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": container with ID starting with 3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336 not found: ID does not exist" Jul 06 22:30:13.902866 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.902785 2580 scope.go:117] "RemoveContainer" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" Jul 06 22:30:13.903225 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:13.903060 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": container with ID starting with 0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc not found: ID does not exist" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" Jul 06 22:30:13.903225 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903092 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc"} err="failed to get container status \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": rpc error: code = NotFound desc = could not find container \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": container with ID starting with 0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc not found: ID does not exist" Jul 06 22:30:13.903225 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903115 2580 scope.go:117] "RemoveContainer" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" Jul 06 22:30:13.903516 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:13.903477 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": container with ID starting with 58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20 not found: ID does not exist" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" Jul 06 22:30:13.903516 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903509 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20"} err="failed to get container status \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": rpc error: code = NotFound desc = could not find container \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": container with ID starting with 58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20 not found: ID does not exist" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903529 2580 scope.go:117] "RemoveContainer" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903531 2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903668 2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903725 2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-grpc-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903741 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-db\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903755 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-tls-assets\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903797 2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-metrics-client-certs\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903812 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903826 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.903835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903841 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903878 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-metrics-client-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903904 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903918 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhzw2\" (UniqueName: \"kubernetes.io/projected/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-kube-api-access-xhzw2\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:13.903873 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": container with ID starting with 0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59 not found: ID does not exist" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903973 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59"} err="failed to get container status \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": rpc error: code = NotFound desc = could not find container \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": container with ID starting with 0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59 not found: ID does not exist" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.903992 2580 scope.go:117] "RemoveContainer" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904078 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904092 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-config-out\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904106 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904118 2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-secret-kube-rbac-proxy\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904134 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7ee67a-77a4-42e4-a6f3-c89df31a13f3-web-config\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:30:13.904811 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:13.904635 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": container with ID starting with cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f not found: ID does not exist" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" Jul 06 22:30:13.904811 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904663 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f"} err="failed to get container status \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": rpc error: code = NotFound desc = could not find container \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": container with ID starting with cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f not found: ID does not exist" Jul 06 22:30:13.904811 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904683 2580 scope.go:117] "RemoveContainer" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" Jul 06 22:30:13.904955 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:30:13.904896 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": container with ID starting with 9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a not found: ID does not exist" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" Jul 06 22:30:13.904955 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904916 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a"} err="failed to get container status \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": rpc error: code = NotFound desc = could not find container \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": container with ID starting with 9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a not found: ID does not exist" Jul 06 22:30:13.904955 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.904931 2580 scope.go:117] "RemoveContainer" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" Jul 06 22:30:13.905202 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.905159 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3"} err="failed to get container status \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": rpc error: code = NotFound desc = could not find container \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": container with ID starting with 282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3 not found: ID does not exist" Jul 06 22:30:13.905261 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.905202 2580 scope.go:117] "RemoveContainer" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" Jul 06 22:30:13.905407 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.905391 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336"} err="failed to get container status \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": rpc error: code = NotFound desc = could not find container \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": container with ID starting with 3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336 not found: ID does not exist" Jul 06 22:30:13.905470 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.905408 2580 scope.go:117] "RemoveContainer" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" Jul 06 22:30:13.905635 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.905617 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc"} err="failed to get container status \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": rpc error: code = NotFound desc = could not find container \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": container with ID starting with 0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc not found: ID does not exist" Jul 06 22:30:13.905696 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.905634 2580 scope.go:117] "RemoveContainer" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" Jul 06 22:30:13.905859 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.905842 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20"} err="failed to get container status \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": rpc error: code = NotFound desc = could not find container \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": container with ID starting with 58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20 not found: ID does not exist" Jul 06 22:30:13.905915 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.905859 2580 scope.go:117] "RemoveContainer" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" Jul 06 22:30:13.906063 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906047 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59"} err="failed to get container status \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": rpc error: code = NotFound desc = could not find container \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": container with ID starting with 0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59 not found: ID does not exist" Jul 06 22:30:13.906121 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906062 2580 scope.go:117] "RemoveContainer" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" Jul 06 22:30:13.906281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906259 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:30:13.906359 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906310 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f"} err="failed to get container status \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": rpc error: code = NotFound desc = could not find container \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": container with ID starting with cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f not found: ID does not exist" Jul 06 22:30:13.906359 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906330 2580 scope.go:117] "RemoveContainer" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" Jul 06 22:30:13.906655 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906609 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a"} err="failed to get container status \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": rpc error: code = NotFound desc = could not find container \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": container with ID starting with 9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a not found: ID does not exist" Jul 06 22:30:13.906727 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906657 2580 scope.go:117] "RemoveContainer" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" Jul 06 22:30:13.906785 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906732 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="thanos-sidecar" Jul 06 22:30:13.906785 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906751 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="thanos-sidecar" Jul 06 22:30:13.906785 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906770 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="prometheus" Jul 06 22:30:13.906785 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906778 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="prometheus" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906791 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="init-config-reloader" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906799 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="init-config-reloader" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906810 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="config-reloader" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906819 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="config-reloader" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906839 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy-thanos" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906847 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy-thanos" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906859 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy-web" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906868 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy-web" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906877 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906885 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906913 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3"} err="failed to get container status \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": rpc error: code = NotFound desc = could not find container \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": container with ID starting with 282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3 not found: ID does not exist" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906935 2580 scope.go:117] "RemoveContainer" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906961 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy" Jul 06 22:30:13.906966 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906972 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy-web" Jul 06 22:30:13.907575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906984 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="kube-rbac-proxy-thanos" Jul 06 22:30:13.907575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.906992 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="prometheus" Jul 06 22:30:13.907575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.907000 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="thanos-sidecar" Jul 06 22:30:13.907575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.907011 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" containerName="config-reloader" Jul 06 22:30:13.907575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.907495 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336"} err="failed to get container status \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": rpc error: code = NotFound desc = could not find container \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": container with ID starting with 3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336 not found: ID does not exist" Jul 06 22:30:13.907575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.907512 2580 scope.go:117] "RemoveContainer" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" Jul 06 22:30:13.907791 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.907773 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc"} err="failed to get container status \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": rpc error: code = NotFound desc = could not find container \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": container with ID starting with 0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc not found: ID does not exist" Jul 06 22:30:13.907828 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.907791 2580 scope.go:117] "RemoveContainer" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" Jul 06 22:30:13.908049 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908031 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20"} err="failed to get container status \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": rpc error: code = NotFound desc = could not find container \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": container with ID starting with 58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20 not found: ID does not exist" Jul 06 22:30:13.908093 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908050 2580 scope.go:117] "RemoveContainer" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" Jul 06 22:30:13.908288 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908270 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59"} err="failed to get container status \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": rpc error: code = NotFound desc = could not find container \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": container with ID starting with 0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59 not found: ID does not exist" Jul 06 22:30:13.908288 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908287 2580 scope.go:117] "RemoveContainer" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" Jul 06 22:30:13.908496 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908479 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f"} err="failed to get container status \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": rpc error: code = NotFound desc = could not find container \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": container with ID starting with cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f not found: ID does not exist" Jul 06 22:30:13.908539 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908496 2580 scope.go:117] "RemoveContainer" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" Jul 06 22:30:13.908685 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908670 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a"} err="failed to get container status \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": rpc error: code = NotFound desc = could not find container \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": container with ID starting with 9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a not found: ID does not exist" Jul 06 22:30:13.908685 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908684 2580 scope.go:117] "RemoveContainer" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" Jul 06 22:30:13.908888 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908873 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3"} err="failed to get container status \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": rpc error: code = NotFound desc = could not find container \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": container with ID starting with 282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3 not found: ID does not exist" Jul 06 22:30:13.908888 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.908888 2580 scope.go:117] "RemoveContainer" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" Jul 06 22:30:13.909093 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909079 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336"} err="failed to get container status \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": rpc error: code = NotFound desc = could not find container \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": container with ID starting with 3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336 not found: ID does not exist" Jul 06 22:30:13.909093 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909093 2580 scope.go:117] "RemoveContainer" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" Jul 06 22:30:13.909310 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909291 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc"} err="failed to get container status \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": rpc error: code = NotFound desc = could not find container \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": container with ID starting with 0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc not found: ID does not exist" Jul 06 22:30:13.909359 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909312 2580 scope.go:117] "RemoveContainer" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" Jul 06 22:30:13.909543 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909523 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20"} err="failed to get container status \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": rpc error: code = NotFound desc = could not find container \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": container with ID starting with 58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20 not found: ID does not exist" Jul 06 22:30:13.909662 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909544 2580 scope.go:117] "RemoveContainer" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" Jul 06 22:30:13.909814 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909798 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59"} err="failed to get container status \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": rpc error: code = NotFound desc = could not find container \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": container with ID starting with 0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59 not found: ID does not exist" Jul 06 22:30:13.909874 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909815 2580 scope.go:117] "RemoveContainer" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" Jul 06 22:30:13.910002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.909983 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f"} err="failed to get container status \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": rpc error: code = NotFound desc = could not find container \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": container with ID starting with cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f not found: ID does not exist" Jul 06 22:30:13.910040 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910003 2580 scope.go:117] "RemoveContainer" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" Jul 06 22:30:13.910202 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910186 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a"} err="failed to get container status \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": rpc error: code = NotFound desc = could not find container \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": container with ID starting with 9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a not found: ID does not exist" Jul 06 22:30:13.910253 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910202 2580 scope.go:117] "RemoveContainer" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" Jul 06 22:30:13.910409 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910392 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3"} err="failed to get container status \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": rpc error: code = NotFound desc = could not find container \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": container with ID starting with 282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3 not found: ID does not exist" Jul 06 22:30:13.910448 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910409 2580 scope.go:117] "RemoveContainer" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" Jul 06 22:30:13.910638 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910620 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336"} err="failed to get container status \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": rpc error: code = NotFound desc = could not find container \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": container with ID starting with 3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336 not found: ID does not exist" Jul 06 22:30:13.910687 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910639 2580 scope.go:117] "RemoveContainer" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" Jul 06 22:30:13.910841 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910824 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc"} err="failed to get container status \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": rpc error: code = NotFound desc = could not find container \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": container with ID starting with 0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc not found: ID does not exist" Jul 06 22:30:13.910901 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910841 2580 scope.go:117] "RemoveContainer" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" Jul 06 22:30:13.911011 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.910998 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20"} err="failed to get container status \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": rpc error: code = NotFound desc = could not find container \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": container with ID starting with 58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20 not found: ID does not exist" Jul 06 22:30:13.911051 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.911012 2580 scope.go:117] "RemoveContainer" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" Jul 06 22:30:13.911149 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.911132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:13.911251 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.911217 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59"} err="failed to get container status \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": rpc error: code = NotFound desc = could not find container \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": container with ID starting with 0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59 not found: ID does not exist" Jul 06 22:30:13.911251 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.911234 2580 scope.go:117] "RemoveContainer" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" Jul 06 22:30:13.911602 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.911454 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f"} err="failed to get container status \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": rpc error: code = NotFound desc = could not find container \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": container with ID starting with cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f not found: ID does not exist" Jul 06 22:30:13.911602 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.911491 2580 scope.go:117] "RemoveContainer" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" Jul 06 22:30:13.911820 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.911795 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a"} err="failed to get container status \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": rpc error: code = NotFound desc = could not find container \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": container with ID starting with 9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a not found: ID does not exist" Jul 06 22:30:13.911868 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.911826 2580 scope.go:117] "RemoveContainer" containerID="282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.914226 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.914509 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.914562 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1mvt7lfvfme4k\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.914570 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.914755 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.914943 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.915013 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.915234 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.915528 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.915662 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-twx67\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.915869 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.916021 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.916202 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 22:30:13.916914 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.916332 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 22:30:13.918587 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.918224 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3"} err="failed to get container status \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": rpc error: code = NotFound desc = could not find container \"282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3\": container with ID starting with 282f6acf924eab294c5e07326f0ddf7035eed64abc44df3a4448b10dc5b4cda3 not found: ID does not exist" Jul 06 22:30:13.918587 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.918297 2580 scope.go:117] "RemoveContainer" containerID="3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336" Jul 06 22:30:13.918933 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.918910 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336"} err="failed to get container status \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": rpc error: code = NotFound desc = could not find container \"3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336\": container with ID starting with 3bb3791302350e40ba54a4dab92ed57a155b5c75b6be99db3c025d53fa69f336 not found: ID does not exist" Jul 06 22:30:13.918933 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.918931 2580 scope.go:117] "RemoveContainer" containerID="0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc" Jul 06 22:30:13.919458 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.919424 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc"} err="failed to get container status \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": rpc error: code = NotFound desc = could not find container \"0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc\": container with ID starting with 0eb9bc0a399eccb4d60094992cc4c7d9c90c2bc950566051587cbbf608d7aedc not found: ID does not exist" Jul 06 22:30:13.919573 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.919459 2580 scope.go:117] "RemoveContainer" containerID="58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20" Jul 06 22:30:13.919573 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.919503 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 22:30:13.919780 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.919756 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20"} err="failed to get container status \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": rpc error: code = NotFound desc = could not find container \"58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20\": container with ID starting with 58b46be08912c94fac81a5695428210c32bbde87e243181b95f2f29e8b078f20 not found: ID does not exist" Jul 06 22:30:13.919842 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.919781 2580 scope.go:117] "RemoveContainer" containerID="0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59" Jul 06 22:30:13.920060 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.920034 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59"} err="failed to get container status \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": rpc error: code = NotFound desc = could not find container \"0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59\": container with ID starting with 0f39f97bdb4f3c045c16dfd91e8fec8330466996088d4896c9d7e09d31e70d59 not found: ID does not exist" Jul 06 22:30:13.920060 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.920059 2580 scope.go:117] "RemoveContainer" containerID="cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f" Jul 06 22:30:13.920372 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.920348 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f"} err="failed to get container status \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": rpc error: code = NotFound desc = could not find container \"cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f\": container with ID starting with cf5fea2bdf9cd6f09e19c34fede36db4c80e5cd5e654217f6c29833b430d560f not found: ID does not exist" Jul 06 22:30:13.920372 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.920369 2580 scope.go:117] "RemoveContainer" containerID="9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a" Jul 06 22:30:13.920640 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.920621 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a"} err="failed to get container status \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": rpc error: code = NotFound desc = could not find container \"9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a\": container with ID starting with 9416c61197c97eed44dc7a19cf6b5bacc539587ad013718a6d91fb2728a5ac3a not found: ID does not exist" Jul 06 22:30:13.921233 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.921215 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 22:30:13.928155 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:13.928133 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:30:14.004873 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.004793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.004873 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.004830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.004873 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.004850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psfx9\" (UniqueName: \"kubernetes.io/projected/9b7f53a6-4aa2-47f4-afe7-8166466e3758-kube-api-access-psfx9\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.004873 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.004871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.004937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b7f53a6-4aa2-47f4-afe7-8166466e3758-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.004977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.004999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b7f53a6-4aa2-47f4-afe7-8166466e3758-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-config\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.005478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.005424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106157 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-config\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106157 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106379 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106379 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106379 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106259 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psfx9\" (UniqueName: \"kubernetes.io/projected/9b7f53a6-4aa2-47f4-afe7-8166466e3758-kube-api-access-psfx9\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106379 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106379 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b7f53a6-4aa2-47f4-afe7-8166466e3758-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106613 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.106613 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.106461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107142 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107118 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107118 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107216 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b7f53a6-4aa2-47f4-afe7-8166466e3758-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107577 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107577 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107577 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107577 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107577 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107577 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107577 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.107987 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.107851 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.108386 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.108060 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.108749 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.108721 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.109443 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.109088 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b7f53a6-4aa2-47f4-afe7-8166466e3758-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.109443 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.109283 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-config\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.109443 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.109409 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.109443 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.109411 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.111474 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.109930 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.111474 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.109948 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.111474 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.110296 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.111474 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.110759 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.111474 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.111119 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b7f53a6-4aa2-47f4-afe7-8166466e3758-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.111809 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.111782 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.112087 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.112064 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.112197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.112072 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b7f53a6-4aa2-47f4-afe7-8166466e3758-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.113108 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.113086 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b7f53a6-4aa2-47f4-afe7-8166466e3758-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.113907 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.113891 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psfx9\" (UniqueName: \"kubernetes.io/projected/9b7f53a6-4aa2-47f4-afe7-8166466e3758-kube-api-access-psfx9\") pod \"prometheus-k8s-0\" (UID: \"9b7f53a6-4aa2-47f4-afe7-8166466e3758\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.228835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.228800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:30:14.366046 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.366023 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:30:14.368414 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:30:14.368386 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7f53a6_4aa2_47f4_afe7_8166466e3758.slice/crio-d9000e8eb240dea9a8274e181376a2064be0aad5c9fa60c2754865f6646f66f3 WatchSource:0}: Error finding container d9000e8eb240dea9a8274e181376a2064be0aad5c9fa60c2754865f6646f66f3: Status 404 returned error can't find the container with id d9000e8eb240dea9a8274e181376a2064be0aad5c9fa60c2754865f6646f66f3 Jul 06 22:30:14.858058 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.858028 2580 generic.go:356] "Generic (PLEG): container finished" podID="9b7f53a6-4aa2-47f4-afe7-8166466e3758" containerID="847c7ac9d6bd8a5ff6bcaf174461ef0c8be4e38789d474b6282a76a866946dad" exitCode=0 Jul 06 22:30:14.858451 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.858079 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b7f53a6-4aa2-47f4-afe7-8166466e3758","Type":"ContainerDied","Data":"847c7ac9d6bd8a5ff6bcaf174461ef0c8be4e38789d474b6282a76a866946dad"} Jul 06 22:30:14.858451 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:14.858127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b7f53a6-4aa2-47f4-afe7-8166466e3758","Type":"ContainerStarted","Data":"d9000e8eb240dea9a8274e181376a2064be0aad5c9fa60c2754865f6646f66f3"} Jul 06 22:30:15.014369 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:15.014329 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af7ee67a-77a4-42e4-a6f3-c89df31a13f3" path="/var/lib/kubelet/pods/af7ee67a-77a4-42e4-a6f3-c89df31a13f3/volumes" Jul 06 22:30:15.866035 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:15.866001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b7f53a6-4aa2-47f4-afe7-8166466e3758","Type":"ContainerStarted","Data":"0c5d0589418f8c0c91ae17fa10f5f7a1af2701629630644523c4fca5f5e5d1ac"} Jul 06 22:30:15.866446 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:15.866064 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b7f53a6-4aa2-47f4-afe7-8166466e3758","Type":"ContainerStarted","Data":"c6cb7a2f5fbe7569fd9981788ad384372189d105fd40ea2d8f0c6ea18cc1ebe2"} Jul 06 22:30:15.866446 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:15.866090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b7f53a6-4aa2-47f4-afe7-8166466e3758","Type":"ContainerStarted","Data":"bd7099b0a33f4fe53a1d3ea4728e525949980f65e4793c7ed018d1485c9e6058"} Jul 06 22:30:15.866446 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:15.866115 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b7f53a6-4aa2-47f4-afe7-8166466e3758","Type":"ContainerStarted","Data":"5cb034f0cd4b01ad0da673d6ddb568203fd0425da52096aa0f1d8ead2cb29fc2"} Jul 06 22:30:15.866446 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:15.866138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b7f53a6-4aa2-47f4-afe7-8166466e3758","Type":"ContainerStarted","Data":"ed7980832fc3805ed8623519757d2f988df8b97ac1ccb6726a3ddfd311341682"} Jul 06 22:30:15.866446 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:15.866207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b7f53a6-4aa2-47f4-afe7-8166466e3758","Type":"ContainerStarted","Data":"7b2a4523896dc6d04f2748511c799d4a94e1dea7813bfbf780ac9725161cb649"} Jul 06 22:30:15.895223 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:15.895149 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.872118489 podStartE2EDuration="2.89513346s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.014971ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:15.893468957 +0000 UTC m=+151.503470168" watchObservedRunningTime="2026-07-06 22:30:15.89513346 +0000 UTC m=+151.505134668" Jul 06 22:30:19.229946 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:30:19.229898 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:31:11.357536 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.357504 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fr9j6"] Jul 06 22:31:11.359742 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.359727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:11.362235 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.362214 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 22:31:11.363155 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.363141 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 22:31:11.363155 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.363148 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gwcz7\"" Jul 06 22:31:11.368485 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.368465 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fr9j6"] Jul 06 22:31:11.526295 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.526256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf39dbea-d4a2-4790-871a-7fabf41e3108-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fr9j6\" (UID: \"bf39dbea-d4a2-4790-871a-7fabf41e3108\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:11.526464 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.526357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcv9z\" (UniqueName: \"kubernetes.io/projected/bf39dbea-d4a2-4790-871a-7fabf41e3108-kube-api-access-dcv9z\") pod \"cert-manager-webhook-6486ffd997-fr9j6\" (UID: \"bf39dbea-d4a2-4790-871a-7fabf41e3108\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:11.544070 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.544040 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p48hx"] Jul 06 22:31:11.546350 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.546334 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" Jul 06 22:31:11.548767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.548750 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wbh9n\"" Jul 06 22:31:11.557087 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.557067 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p48hx"] Jul 06 22:31:11.627279 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.627193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf39dbea-d4a2-4790-871a-7fabf41e3108-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fr9j6\" (UID: \"bf39dbea-d4a2-4790-871a-7fabf41e3108\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:11.627279 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.627253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481fdc1d-172f-45e8-85a3-b4f78f2ec743-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p48hx\" (UID: \"481fdc1d-172f-45e8-85a3-b4f78f2ec743\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" Jul 06 22:31:11.627509 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.627338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6ds\" (UniqueName: \"kubernetes.io/projected/481fdc1d-172f-45e8-85a3-b4f78f2ec743-kube-api-access-ww6ds\") pod \"cert-manager-cainjector-559cdd8c87-p48hx\" (UID: \"481fdc1d-172f-45e8-85a3-b4f78f2ec743\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" Jul 06 22:31:11.627509 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.627422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcv9z\" (UniqueName: \"kubernetes.io/projected/bf39dbea-d4a2-4790-871a-7fabf41e3108-kube-api-access-dcv9z\") pod \"cert-manager-webhook-6486ffd997-fr9j6\" (UID: \"bf39dbea-d4a2-4790-871a-7fabf41e3108\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:11.635719 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.635689 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf39dbea-d4a2-4790-871a-7fabf41e3108-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fr9j6\" (UID: \"bf39dbea-d4a2-4790-871a-7fabf41e3108\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:11.635825 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.635769 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcv9z\" (UniqueName: \"kubernetes.io/projected/bf39dbea-d4a2-4790-871a-7fabf41e3108-kube-api-access-dcv9z\") pod \"cert-manager-webhook-6486ffd997-fr9j6\" (UID: \"bf39dbea-d4a2-4790-871a-7fabf41e3108\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:11.681671 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.681641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:11.728982 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.728946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481fdc1d-172f-45e8-85a3-b4f78f2ec743-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p48hx\" (UID: \"481fdc1d-172f-45e8-85a3-b4f78f2ec743\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" Jul 06 22:31:11.729158 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.729012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6ds\" (UniqueName: \"kubernetes.io/projected/481fdc1d-172f-45e8-85a3-b4f78f2ec743-kube-api-access-ww6ds\") pod \"cert-manager-cainjector-559cdd8c87-p48hx\" (UID: \"481fdc1d-172f-45e8-85a3-b4f78f2ec743\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" Jul 06 22:31:11.737290 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.737266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481fdc1d-172f-45e8-85a3-b4f78f2ec743-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p48hx\" (UID: \"481fdc1d-172f-45e8-85a3-b4f78f2ec743\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" Jul 06 22:31:11.737413 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.737364 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6ds\" (UniqueName: \"kubernetes.io/projected/481fdc1d-172f-45e8-85a3-b4f78f2ec743-kube-api-access-ww6ds\") pod \"cert-manager-cainjector-559cdd8c87-p48hx\" (UID: \"481fdc1d-172f-45e8-85a3-b4f78f2ec743\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" Jul 06 22:31:11.808714 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.808622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fr9j6"] Jul 06 22:31:11.811439 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:31:11.811412 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf39dbea_d4a2_4790_871a_7fabf41e3108.slice/crio-12fea9a9ac3cc9c389fd217bcd797a35e01e227f10154307021f6aa2c9231ab9 WatchSource:0}: Error finding container 12fea9a9ac3cc9c389fd217bcd797a35e01e227f10154307021f6aa2c9231ab9: Status 404 returned error can't find the container with id 12fea9a9ac3cc9c389fd217bcd797a35e01e227f10154307021f6aa2c9231ab9 Jul 06 22:31:11.855304 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.855265 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" Jul 06 22:31:11.983223 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:11.983192 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p48hx"] Jul 06 22:31:11.985457 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:31:11.985434 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod481fdc1d_172f_45e8_85a3_b4f78f2ec743.slice/crio-519c180115f2f63c90e554d67fb472d47cd48d698799a52fc29a85c659563805 WatchSource:0}: Error finding container 519c180115f2f63c90e554d67fb472d47cd48d698799a52fc29a85c659563805: Status 404 returned error can't find the container with id 519c180115f2f63c90e554d67fb472d47cd48d698799a52fc29a85c659563805 Jul 06 22:31:12.098199 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:12.098141 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" event={"ID":"bf39dbea-d4a2-4790-871a-7fabf41e3108","Type":"ContainerStarted","Data":"12fea9a9ac3cc9c389fd217bcd797a35e01e227f10154307021f6aa2c9231ab9"} Jul 06 22:31:12.099312 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:12.099288 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" event={"ID":"481fdc1d-172f-45e8-85a3-b4f78f2ec743","Type":"ContainerStarted","Data":"519c180115f2f63c90e554d67fb472d47cd48d698799a52fc29a85c659563805"} Jul 06 22:31:14.229543 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:14.229511 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:31:14.248119 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:14.248084 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:31:15.120305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:15.120260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" event={"ID":"bf39dbea-d4a2-4790-871a-7fabf41e3108","Type":"ContainerStarted","Data":"6eb4ff66146a71f27a7544878a8fb3c4f75f55964fd338b2d4aac61133549e3d"} Jul 06 22:31:15.120538 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:15.120514 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:15.121931 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:15.121905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" event={"ID":"481fdc1d-172f-45e8-85a3-b4f78f2ec743","Type":"ContainerStarted","Data":"c336818135fdcc67f57be860cf81d63a0b8edddf02714049efea0de14916f6a0"} Jul 06 22:31:15.138974 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:15.138943 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:31:15.152129 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:15.152083 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" podStartSLOduration=0.963785297 podStartE2EDuration="4.152066621s" totalImagesPullingTime="3.188281324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:15.151620034 +0000 UTC m=+210.761621244" watchObservedRunningTime="2026-07-06 22:31:15.152066621 +0000 UTC m=+210.762067831" Jul 06 22:31:15.203574 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:15.203513 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p48hx" podStartSLOduration=1.198924369 podStartE2EDuration="4.203489511s" totalImagesPullingTime="3.004565142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:15.202465645 +0000 UTC m=+210.812466873" watchObservedRunningTime="2026-07-06 22:31:15.203489511 +0000 UTC m=+210.813490720" Jul 06 22:31:20.935271 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:20.935230 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pwf9p"] Jul 06 22:31:20.937517 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:20.937501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pwf9p" Jul 06 22:31:20.943986 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:20.943786 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-q5np6\"" Jul 06 22:31:20.954732 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:20.954706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pwf9p"] Jul 06 22:31:21.117555 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.117515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsk9v\" (UniqueName: \"kubernetes.io/projected/2acd8574-1930-4c36-9395-1b879c293194-kube-api-access-zsk9v\") pod \"cert-manager-55b4d954d5-pwf9p\" (UID: \"2acd8574-1930-4c36-9395-1b879c293194\") " pod="cert-manager/cert-manager-55b4d954d5-pwf9p" Jul 06 22:31:21.117734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.117564 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2acd8574-1930-4c36-9395-1b879c293194-bound-sa-token\") pod \"cert-manager-55b4d954d5-pwf9p\" (UID: \"2acd8574-1930-4c36-9395-1b879c293194\") " pod="cert-manager/cert-manager-55b4d954d5-pwf9p" Jul 06 22:31:21.128244 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.128222 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fr9j6" Jul 06 22:31:21.218799 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.218709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsk9v\" (UniqueName: \"kubernetes.io/projected/2acd8574-1930-4c36-9395-1b879c293194-kube-api-access-zsk9v\") pod \"cert-manager-55b4d954d5-pwf9p\" (UID: \"2acd8574-1930-4c36-9395-1b879c293194\") " pod="cert-manager/cert-manager-55b4d954d5-pwf9p" Jul 06 22:31:21.218799 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.218743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2acd8574-1930-4c36-9395-1b879c293194-bound-sa-token\") pod \"cert-manager-55b4d954d5-pwf9p\" (UID: \"2acd8574-1930-4c36-9395-1b879c293194\") " pod="cert-manager/cert-manager-55b4d954d5-pwf9p" Jul 06 22:31:21.226242 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.226220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2acd8574-1930-4c36-9395-1b879c293194-bound-sa-token\") pod \"cert-manager-55b4d954d5-pwf9p\" (UID: \"2acd8574-1930-4c36-9395-1b879c293194\") " pod="cert-manager/cert-manager-55b4d954d5-pwf9p" Jul 06 22:31:21.226427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.226411 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsk9v\" (UniqueName: \"kubernetes.io/projected/2acd8574-1930-4c36-9395-1b879c293194-kube-api-access-zsk9v\") pod \"cert-manager-55b4d954d5-pwf9p\" (UID: \"2acd8574-1930-4c36-9395-1b879c293194\") " pod="cert-manager/cert-manager-55b4d954d5-pwf9p" Jul 06 22:31:21.246457 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.246428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pwf9p" Jul 06 22:31:21.379834 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:31:21.379802 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2acd8574_1930_4c36_9395_1b879c293194.slice/crio-1bfcc7158a07deea7e6f4585af25ed46e9e2c3f27ee99a24e207613cd1ce34cd WatchSource:0}: Error finding container 1bfcc7158a07deea7e6f4585af25ed46e9e2c3f27ee99a24e207613cd1ce34cd: Status 404 returned error can't find the container with id 1bfcc7158a07deea7e6f4585af25ed46e9e2c3f27ee99a24e207613cd1ce34cd Jul 06 22:31:21.386288 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:21.383146 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pwf9p"] Jul 06 22:31:22.153699 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:22.153667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pwf9p" event={"ID":"2acd8574-1930-4c36-9395-1b879c293194","Type":"ContainerStarted","Data":"3c027e5b43d193b6e9b13e4972dc862393b45df6069c7d6bf22593f55c48d5d3"} Jul 06 22:31:22.154080 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:22.153707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pwf9p" event={"ID":"2acd8574-1930-4c36-9395-1b879c293194","Type":"ContainerStarted","Data":"1bfcc7158a07deea7e6f4585af25ed46e9e2c3f27ee99a24e207613cd1ce34cd"} Jul 06 22:31:22.172540 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:22.172485 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pwf9p" podStartSLOduration=2.172469761 podStartE2EDuration="2.172469761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:22.170457677 +0000 UTC m=+217.780458900" watchObservedRunningTime="2026-07-06 22:31:22.172469761 +0000 UTC m=+217.782470970" Jul 06 22:31:46.439241 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.439210 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z"] Jul 06 22:31:46.441661 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.441645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.444671 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.444652 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 22:31:46.445073 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.445052 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 22:31:46.445201 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.445095 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 22:31:46.446049 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.446021 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wb5lr\"" Jul 06 22:31:46.446297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.446274 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 22:31:46.457144 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.457123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z"] Jul 06 22:31:46.534687 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.534655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2725-f46b-40b6-9516-0374fe569eb6-manifests\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.534687 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.534690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce9a2725-f46b-40b6-9516-0374fe569eb6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.534923 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.534708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6c9\" (UniqueName: \"kubernetes.io/projected/ce9a2725-f46b-40b6-9516-0374fe569eb6-kube-api-access-jv6c9\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.534923 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.534777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2725-f46b-40b6-9516-0374fe569eb6-tmp\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.534923 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.534885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce9a2725-f46b-40b6-9516-0374fe569eb6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.636219 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.636153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2725-f46b-40b6-9516-0374fe569eb6-manifests\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.636219 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.636213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce9a2725-f46b-40b6-9516-0374fe569eb6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.636418 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.636328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6c9\" (UniqueName: \"kubernetes.io/projected/ce9a2725-f46b-40b6-9516-0374fe569eb6-kube-api-access-jv6c9\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.636477 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.636458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2725-f46b-40b6-9516-0374fe569eb6-tmp\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.636576 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.636559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce9a2725-f46b-40b6-9516-0374fe569eb6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.636664 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.636611 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2725-f46b-40b6-9516-0374fe569eb6-manifests\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.638687 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.638658 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2725-f46b-40b6-9516-0374fe569eb6-tmp\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.638793 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.638728 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce9a2725-f46b-40b6-9516-0374fe569eb6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.639234 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.639214 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce9a2725-f46b-40b6-9516-0374fe569eb6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.644688 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.644661 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6c9\" (UniqueName: \"kubernetes.io/projected/ce9a2725-f46b-40b6-9516-0374fe569eb6-kube-api-access-jv6c9\") pod \"opendatahub-operator-controller-manager-8c9b4d57b-wcp4z\" (UID: \"ce9a2725-f46b-40b6-9516-0374fe569eb6\") " pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.752377 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.752284 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:46.904462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:46.904431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z"] Jul 06 22:31:46.908060 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:31:46.908030 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9a2725_f46b_40b6_9516_0374fe569eb6.slice/crio-2f1cee01179a583dac67f7f38ab814a6aedfdefea7f3f56b421be050ee333d7c WatchSource:0}: Error finding container 2f1cee01179a583dac67f7f38ab814a6aedfdefea7f3f56b421be050ee333d7c: Status 404 returned error can't find the container with id 2f1cee01179a583dac67f7f38ab814a6aedfdefea7f3f56b421be050ee333d7c Jul 06 22:31:47.256120 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:47.256086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" event={"ID":"ce9a2725-f46b-40b6-9516-0374fe569eb6","Type":"ContainerStarted","Data":"2f1cee01179a583dac67f7f38ab814a6aedfdefea7f3f56b421be050ee333d7c"} Jul 06 22:31:50.271645 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:50.271607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" event={"ID":"ce9a2725-f46b-40b6-9516-0374fe569eb6","Type":"ContainerStarted","Data":"5df722fac4881f846eff22d5ad95b90834c010c5c9bf37d3c4e2f50512a2d76d"} Jul 06 22:31:51.277364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:51.277330 2580 generic.go:356] "Generic (PLEG): container finished" podID="ce9a2725-f46b-40b6-9516-0374fe569eb6" containerID="5df722fac4881f846eff22d5ad95b90834c010c5c9bf37d3c4e2f50512a2d76d" exitCode=0 Jul 06 22:31:51.277765 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:51.277392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" event={"ID":"ce9a2725-f46b-40b6-9516-0374fe569eb6","Type":"ContainerDied","Data":"5df722fac4881f846eff22d5ad95b90834c010c5c9bf37d3c4e2f50512a2d76d"} Jul 06 22:31:52.284176 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:52.284131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" event={"ID":"ce9a2725-f46b-40b6-9516-0374fe569eb6","Type":"ContainerStarted","Data":"c7b960a22c23bf5a64c7655b1dce101c71f74269bc9f71361899fcc60ddc11ad"} Jul 06 22:31:52.284580 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:52.284279 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:31:52.307461 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:31:52.307414 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" podStartSLOduration=2.127975353 podStartE2EDuration="6.307400758s" totalImagesPullingTime="3.158504384s" totalInitContainerRuntime="1.020921021s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:52.307188022 +0000 UTC m=+247.917189222" watchObservedRunningTime="2026-07-06 22:31:52.307400758 +0000 UTC m=+247.917401968" Jul 06 22:32:03.290474 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:03.290439 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8c9b4d57b-wcp4z" Jul 06 22:32:16.192333 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.192251 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c"] Jul 06 22:32:16.195807 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.195785 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.200563 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.200535 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 22:32:16.200785 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.200766 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 22:32:16.202389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.202119 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 22:32:16.202389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.202126 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 22:32:16.202389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.202204 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-l98fd\"" Jul 06 22:32:16.202389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.202226 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:32:16.213988 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.213966 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c"] Jul 06 22:32:16.301962 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.301925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d51ade3-a1c4-4c70-8069-bc41c464e585-manager-config\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.302150 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.301968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d51ade3-a1c4-4c70-8069-bc41c464e585-metrics-cert\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.302239 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.302187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d51ade3-a1c4-4c70-8069-bc41c464e585-cert\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.302279 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.302234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqzj\" (UniqueName: \"kubernetes.io/projected/3d51ade3-a1c4-4c70-8069-bc41c464e585-kube-api-access-lzqzj\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.403018 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.402983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d51ade3-a1c4-4c70-8069-bc41c464e585-cert\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.403018 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.403021 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqzj\" (UniqueName: \"kubernetes.io/projected/3d51ade3-a1c4-4c70-8069-bc41c464e585-kube-api-access-lzqzj\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.403292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.403066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d51ade3-a1c4-4c70-8069-bc41c464e585-manager-config\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.403292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.403089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d51ade3-a1c4-4c70-8069-bc41c464e585-metrics-cert\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.403795 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.403769 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d51ade3-a1c4-4c70-8069-bc41c464e585-manager-config\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.405394 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.405375 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d51ade3-a1c4-4c70-8069-bc41c464e585-metrics-cert\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.405606 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.405587 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d51ade3-a1c4-4c70-8069-bc41c464e585-cert\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.413092 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.413071 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqzj\" (UniqueName: \"kubernetes.io/projected/3d51ade3-a1c4-4c70-8069-bc41c464e585-kube-api-access-lzqzj\") pod \"lws-controller-manager-c7f89b564-8nd7c\" (UID: \"3d51ade3-a1c4-4c70-8069-bc41c464e585\") " pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.506150 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.506076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:16.653751 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.653719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c"] Jul 06 22:32:16.654985 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:32:16.654962 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d51ade3_a1c4_4c70_8069_bc41c464e585.slice/crio-4da6f02064f1fafda66dd7ae7c81338eee004c70817f8d7b3df7b364266df8fb WatchSource:0}: Error finding container 4da6f02064f1fafda66dd7ae7c81338eee004c70817f8d7b3df7b364266df8fb: Status 404 returned error can't find the container with id 4da6f02064f1fafda66dd7ae7c81338eee004c70817f8d7b3df7b364266df8fb Jul 06 22:32:16.950448 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.950413 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j"] Jul 06 22:32:16.954709 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.954687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:16.957197 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.957158 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 22:32:16.957344 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.957230 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bnt82\"" Jul 06 22:32:16.960760 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.960719 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27"] Jul 06 22:32:16.964255 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.964232 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j"] Jul 06 22:32:16.964375 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.964342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:16.966806 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.966783 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9jbfx\"" Jul 06 22:32:16.966935 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.966849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 22:32:16.978383 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:16.978358 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27"] Jul 06 22:32:17.008289 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.008258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95926\" (UniqueName: \"kubernetes.io/projected/e1a6e50e-38c4-4413-a5ae-3f12c45367b3-kube-api-access-95926\") pod \"llmisvc-controller-manager-777c6bd7d5-p7g27\" (UID: \"e1a6e50e-38c4-4413-a5ae-3f12c45367b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:17.008289 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.008290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1a6e50e-38c4-4413-a5ae-3f12c45367b3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p7g27\" (UID: \"e1a6e50e-38c4-4413-a5ae-3f12c45367b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:17.008496 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.008311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-cert\") pod \"kserve-controller-manager-d6f4b65b8-k7h5j\" (UID: \"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:17.008496 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.008326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874ph\" (UniqueName: \"kubernetes.io/projected/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-kube-api-access-874ph\") pod \"kserve-controller-manager-d6f4b65b8-k7h5j\" (UID: \"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:17.108878 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.108844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95926\" (UniqueName: \"kubernetes.io/projected/e1a6e50e-38c4-4413-a5ae-3f12c45367b3-kube-api-access-95926\") pod \"llmisvc-controller-manager-777c6bd7d5-p7g27\" (UID: \"e1a6e50e-38c4-4413-a5ae-3f12c45367b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:17.108878 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.108882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1a6e50e-38c4-4413-a5ae-3f12c45367b3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p7g27\" (UID: \"e1a6e50e-38c4-4413-a5ae-3f12c45367b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:17.109087 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.108906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-cert\") pod \"kserve-controller-manager-d6f4b65b8-k7h5j\" (UID: \"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:17.109221 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:32:17.109203 2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 22:32:17.109303 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.109240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874ph\" (UniqueName: \"kubernetes.io/projected/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-kube-api-access-874ph\") pod \"kserve-controller-manager-d6f4b65b8-k7h5j\" (UID: \"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:17.109303 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:32:17.109272 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-cert podName:f8c50d40-06ed-4ad5-9df5-abf9c313e5dc nodeName:}" failed. No retries permitted until 2026-07-06 22:32:17.609250711 +0000 UTC m=+273.219251917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-cert") pod "kserve-controller-manager-d6f4b65b8-k7h5j" (UID: "f8c50d40-06ed-4ad5-9df5-abf9c313e5dc") : secret "kserve-webhook-server-cert" not found Jul 06 22:32:17.111316 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.111300 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1a6e50e-38c4-4413-a5ae-3f12c45367b3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p7g27\" (UID: \"e1a6e50e-38c4-4413-a5ae-3f12c45367b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:17.125500 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.125476 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-874ph\" (UniqueName: \"kubernetes.io/projected/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-kube-api-access-874ph\") pod \"kserve-controller-manager-d6f4b65b8-k7h5j\" (UID: \"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:17.128207 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.128189 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95926\" (UniqueName: \"kubernetes.io/projected/e1a6e50e-38c4-4413-a5ae-3f12c45367b3-kube-api-access-95926\") pod \"llmisvc-controller-manager-777c6bd7d5-p7g27\" (UID: \"e1a6e50e-38c4-4413-a5ae-3f12c45367b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:17.275944 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.275857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:17.399090 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.399061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" event={"ID":"3d51ade3-a1c4-4c70-8069-bc41c464e585","Type":"ContainerStarted","Data":"4da6f02064f1fafda66dd7ae7c81338eee004c70817f8d7b3df7b364266df8fb"} Jul 06 22:32:17.412838 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.412689 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27"] Jul 06 22:32:17.414856 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:32:17.414829 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1a6e50e_38c4_4413_a5ae_3f12c45367b3.slice/crio-4b9042e168f4a14394a38283a8cb7114faa2780dc3ce3f3029cdf5faea87b259 WatchSource:0}: Error finding container 4b9042e168f4a14394a38283a8cb7114faa2780dc3ce3f3029cdf5faea87b259: Status 404 returned error can't find the container with id 4b9042e168f4a14394a38283a8cb7114faa2780dc3ce3f3029cdf5faea87b259 Jul 06 22:32:17.622645 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.622549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-cert\") pod \"kserve-controller-manager-d6f4b65b8-k7h5j\" (UID: \"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:17.624933 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.624907 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c50d40-06ed-4ad5-9df5-abf9c313e5dc-cert\") pod \"kserve-controller-manager-d6f4b65b8-k7h5j\" (UID: \"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:17.867806 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:17.867764 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:18.000863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:18.000840 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j"] Jul 06 22:32:18.003115 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:32:18.003073 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c50d40_06ed_4ad5_9df5_abf9c313e5dc.slice/crio-9e5fc3a691f31a566eae6adf7946a5086fb8a191eb5f9fd3e2a77c99e1ca2863 WatchSource:0}: Error finding container 9e5fc3a691f31a566eae6adf7946a5086fb8a191eb5f9fd3e2a77c99e1ca2863: Status 404 returned error can't find the container with id 9e5fc3a691f31a566eae6adf7946a5086fb8a191eb5f9fd3e2a77c99e1ca2863 Jul 06 22:32:18.404407 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:18.404370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" event={"ID":"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc","Type":"ContainerStarted","Data":"9e5fc3a691f31a566eae6adf7946a5086fb8a191eb5f9fd3e2a77c99e1ca2863"} Jul 06 22:32:18.405575 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:18.405546 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" event={"ID":"e1a6e50e-38c4-4413-a5ae-3f12c45367b3","Type":"ContainerStarted","Data":"4b9042e168f4a14394a38283a8cb7114faa2780dc3ce3f3029cdf5faea87b259"} Jul 06 22:32:22.428552 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.428514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" event={"ID":"f8c50d40-06ed-4ad5-9df5-abf9c313e5dc","Type":"ContainerStarted","Data":"880ca1a1ca84d17d0ccc642a3c3409ca5484e83944a58f4f24b8ab13e45ad545"} Jul 06 22:32:22.429101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.428598 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:32:22.430074 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.430048 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" event={"ID":"e1a6e50e-38c4-4413-a5ae-3f12c45367b3","Type":"ContainerStarted","Data":"9cb4123230949abca534430ed2a90bc0ec1e6f98c442df8aa7fb268bf3bc4845"} Jul 06 22:32:22.430227 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.430110 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:22.431631 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.431608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" event={"ID":"3d51ade3-a1c4-4c70-8069-bc41c464e585","Type":"ContainerStarted","Data":"60f841c4c32c5ebfdba515ceeba6162a4864f3dddead7c36722306cef556085d"} Jul 06 22:32:22.431756 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.431739 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:22.458032 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.457984 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" podStartSLOduration=2.6997231299999997 podStartE2EDuration="6.457969593s" totalImagesPullingTime="3.758246463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:22.457461469 +0000 UTC m=+278.067462679" watchObservedRunningTime="2026-07-06 22:32:22.457969593 +0000 UTC m=+278.067970801" Jul 06 22:32:22.485440 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.485393 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" podStartSLOduration=2.136101459 podStartE2EDuration="6.48537827s" totalImagesPullingTime="4.349276811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:22.483627803 +0000 UTC m=+278.093629013" watchObservedRunningTime="2026-07-06 22:32:22.48537827 +0000 UTC m=+278.095379457" Jul 06 22:32:22.518852 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:22.518805 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" podStartSLOduration=1.4007333229999999 podStartE2EDuration="6.518789704s" totalImagesPullingTime="5.118056381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:22.51806851 +0000 UTC m=+278.128069718" watchObservedRunningTime="2026-07-06 22:32:22.518789704 +0000 UTC m=+278.128790912" Jul 06 22:32:33.438734 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:33.438706 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c7f89b564-8nd7c" Jul 06 22:32:44.831704 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:44.831678 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 22:32:53.438861 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:53.438833 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7g27" Jul 06 22:32:53.442309 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:32:53.442272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k7h5j" Jul 06 22:33:47.270524 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.270489 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g"] Jul 06 22:33:47.274194 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.274175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:47.276523 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.276497 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 22:33:47.276657 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.276543 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jvbzk\"" Jul 06 22:33:47.276657 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.276616 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 22:33:47.286273 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.286249 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g"] Jul 06 22:33:47.330012 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.329973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lqj\" (UniqueName: \"kubernetes.io/projected/10737c7e-4a83-497a-8d3d-3a43748d88b5-kube-api-access-n6lqj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" (UID: \"10737c7e-4a83-497a-8d3d-3a43748d88b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:47.330211 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.330059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10737c7e-4a83-497a-8d3d-3a43748d88b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" (UID: \"10737c7e-4a83-497a-8d3d-3a43748d88b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:47.431085 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.431047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lqj\" (UniqueName: \"kubernetes.io/projected/10737c7e-4a83-497a-8d3d-3a43748d88b5-kube-api-access-n6lqj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" (UID: \"10737c7e-4a83-497a-8d3d-3a43748d88b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:47.431297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.431118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10737c7e-4a83-497a-8d3d-3a43748d88b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" (UID: \"10737c7e-4a83-497a-8d3d-3a43748d88b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:47.431502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.431483 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10737c7e-4a83-497a-8d3d-3a43748d88b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" (UID: \"10737c7e-4a83-497a-8d3d-3a43748d88b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:47.445423 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.445392 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lqj\" (UniqueName: \"kubernetes.io/projected/10737c7e-4a83-497a-8d3d-3a43748d88b5-kube-api-access-n6lqj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" (UID: \"10737c7e-4a83-497a-8d3d-3a43748d88b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:47.585863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.585776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:47.727409 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.727385 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g"] Jul 06 22:33:47.729657 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:33:47.729624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10737c7e_4a83_497a_8d3d_3a43748d88b5.slice/crio-f944ef9ea6169dd0fdcd247ed03253952269932fc63b36af9e72e4119739ce44 WatchSource:0}: Error finding container f944ef9ea6169dd0fdcd247ed03253952269932fc63b36af9e72e4119739ce44: Status 404 returned error can't find the container with id f944ef9ea6169dd0fdcd247ed03253952269932fc63b36af9e72e4119739ce44 Jul 06 22:33:47.732003 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.731983 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:33:47.813450 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:47.813406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" event={"ID":"10737c7e-4a83-497a-8d3d-3a43748d88b5","Type":"ContainerStarted","Data":"f944ef9ea6169dd0fdcd247ed03253952269932fc63b36af9e72e4119739ce44"} Jul 06 22:33:52.841484 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:52.841392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" event={"ID":"10737c7e-4a83-497a-8d3d-3a43748d88b5","Type":"ContainerStarted","Data":"473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5"} Jul 06 22:33:52.841891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:52.841485 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:33:52.863122 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:33:52.863076 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" podStartSLOduration=1.055762765 podStartE2EDuration="5.86306191s" totalImagesPullingTime="4.807299145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:33:52.861859044 +0000 UTC m=+368.471860265" watchObservedRunningTime="2026-07-06 22:33:52.86306191 +0000 UTC m=+368.473063118" Jul 06 22:34:03.848188 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:03.848132 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:04.922998 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:04.922959 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2"] Jul 06 22:34:04.926707 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:04.926691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:04.938124 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:04.938102 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2"] Jul 06 22:34:05.003631 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.003599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" (UID: \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.003797 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.003638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4hx\" (UniqueName: \"kubernetes.io/projected/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-kube-api-access-ml4hx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" (UID: \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.104388 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.104352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" (UID: \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.104388 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.104390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4hx\" (UniqueName: \"kubernetes.io/projected/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-kube-api-access-ml4hx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" (UID: \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.104767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.104744 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" (UID: \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.120788 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.120765 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4hx\" (UniqueName: \"kubernetes.io/projected/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-kube-api-access-ml4hx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" (UID: \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.237142 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.237107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.373780 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.373754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2"] Jul 06 22:34:05.375935 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:34:05.375908 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ab7e06_3ce4_49ce_b7d8_c807c5a69c87.slice/crio-cacbfa133826bf4c0eacc2d95a23d74930d7a1a641b97d17733f2f0fb207ecca WatchSource:0}: Error finding container cacbfa133826bf4c0eacc2d95a23d74930d7a1a641b97d17733f2f0fb207ecca: Status 404 returned error can't find the container with id cacbfa133826bf4c0eacc2d95a23d74930d7a1a641b97d17733f2f0fb207ecca Jul 06 22:34:05.554809 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.554720 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2"] Jul 06 22:34:05.564920 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.564862 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2"] Jul 06 22:34:05.582739 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.582707 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g"] Jul 06 22:34:05.582985 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.582961 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" containerName="manager" containerID="cri-o://473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5" gracePeriod=2 Jul 06 22:34:05.584642 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.584618 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g"] Jul 06 22:34:05.585370 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.585343 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:05.607060 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.607033 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb"] Jul 06 22:34:05.607464 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.607448 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" containerName="manager" Jul 06 22:34:05.607549 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.607467 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" containerName="manager" Jul 06 22:34:05.607549 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.607488 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" containerName="manager" Jul 06 22:34:05.607549 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.607496 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" containerName="manager" Jul 06 22:34:05.607698 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.607596 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" containerName="manager" Jul 06 22:34:05.607698 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.607610 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" containerName="manager" Jul 06 22:34:05.611180 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.611080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:05.613627 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.613600 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:05.633080 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.633050 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7"] Jul 06 22:34:05.636872 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.636848 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:05.637030 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.637005 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb"] Jul 06 22:34:05.662270 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.662247 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7"] Jul 06 22:34:05.683666 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.683630 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:05.709212 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.709150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f6tb7\" (UID: \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:05.709332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.709229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d038e6-db32-49cd-9483-614d24444708-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vwcpb\" (UID: \"b9d038e6-db32-49cd-9483-614d24444708\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:05.709332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.709261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26d5\" (UniqueName: \"kubernetes.io/projected/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-kube-api-access-l26d5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f6tb7\" (UID: \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:05.709469 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.709446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgntd\" (UniqueName: \"kubernetes.io/projected/b9d038e6-db32-49cd-9483-614d24444708-kube-api-access-fgntd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vwcpb\" (UID: \"b9d038e6-db32-49cd-9483-614d24444708\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:05.810069 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.810035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f6tb7\" (UID: \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:05.810224 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.810089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d038e6-db32-49cd-9483-614d24444708-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vwcpb\" (UID: \"b9d038e6-db32-49cd-9483-614d24444708\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:05.810224 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.810134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26d5\" (UniqueName: \"kubernetes.io/projected/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-kube-api-access-l26d5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f6tb7\" (UID: \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:05.810317 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.810240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgntd\" (UniqueName: \"kubernetes.io/projected/b9d038e6-db32-49cd-9483-614d24444708-kube-api-access-fgntd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vwcpb\" (UID: \"b9d038e6-db32-49cd-9483-614d24444708\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:05.810467 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.810445 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f6tb7\" (UID: \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:05.810543 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.810514 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d038e6-db32-49cd-9483-614d24444708-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vwcpb\" (UID: \"b9d038e6-db32-49cd-9483-614d24444708\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:05.815554 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.815538 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:05.817712 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.817689 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:05.824434 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.824410 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26d5\" (UniqueName: \"kubernetes.io/projected/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-kube-api-access-l26d5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f6tb7\" (UID: \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:05.824826 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.824811 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgntd\" (UniqueName: \"kubernetes.io/projected/b9d038e6-db32-49cd-9483-614d24444708-kube-api-access-fgntd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vwcpb\" (UID: \"b9d038e6-db32-49cd-9483-614d24444708\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:05.904608 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.904578 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" container="manager" Jul 06 22:34:05.905835 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.905807 2580 generic.go:356] "Generic (PLEG): container finished" podID="10737c7e-4a83-497a-8d3d-3a43748d88b5" containerID="473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5" exitCode=0 Jul 06 22:34:05.905963 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.905848 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:05.905963 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.905859 2580 scope.go:117] "RemoveContainer" containerID="473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5" Jul 06 22:34:05.906972 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.906905 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:05.908922 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.908898 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.910749 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.910721 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:05.910868 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.910849 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10737c7e-4a83-497a-8d3d-3a43748d88b5-extensions-socket-volume\") pod \"10737c7e-4a83-497a-8d3d-3a43748d88b5\" (UID: \"10737c7e-4a83-497a-8d3d-3a43748d88b5\") " Jul 06 22:34:05.910927 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.910913 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6lqj\" (UniqueName: \"kubernetes.io/projected/10737c7e-4a83-497a-8d3d-3a43748d88b5-kube-api-access-n6lqj\") pod \"10737c7e-4a83-497a-8d3d-3a43748d88b5\" (UID: \"10737c7e-4a83-497a-8d3d-3a43748d88b5\") " Jul 06 22:34:05.911281 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.911259 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10737c7e-4a83-497a-8d3d-3a43748d88b5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "10737c7e-4a83-497a-8d3d-3a43748d88b5" (UID: "10737c7e-4a83-497a-8d3d-3a43748d88b5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:34:05.912463 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.912441 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:05.913181 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.913137 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10737c7e-4a83-497a-8d3d-3a43748d88b5-kube-api-access-n6lqj" (OuterVolumeSpecName: "kube-api-access-n6lqj") pod "10737c7e-4a83-497a-8d3d-3a43748d88b5" (UID: "10737c7e-4a83-497a-8d3d-3a43748d88b5"). InnerVolumeSpecName "kube-api-access-n6lqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:34:05.914674 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.914659 2580 scope.go:117] "RemoveContainer" containerID="473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5" Jul 06 22:34:05.915017 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:05.915000 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5\": container with ID starting with 473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5 not found: ID does not exist" containerID="473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5" Jul 06 22:34:05.915070 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.915025 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5"} err="failed to get container status \"473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5\": rpc error: code = NotFound desc = could not find container \"473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5\": container with ID starting with 473b5d1eefd9205565e37f63cf39558d28f5c6de031e843033c5d2e765ee99f5 not found: ID does not exist" Jul 06 22:34:05.945828 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.945802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:05.951503 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:05.951485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:06.013542 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.013075 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10737c7e-4a83-497a-8d3d-3a43748d88b5-extensions-socket-volume\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:34:06.013542 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.013107 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6lqj\" (UniqueName: \"kubernetes.io/projected/10737c7e-4a83-497a-8d3d-3a43748d88b5-kube-api-access-n6lqj\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:34:06.093230 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.093196 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb"] Jul 06 22:34:06.094807 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:34:06.094769 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d038e6_db32_49cd_9483_614d24444708.slice/crio-4fe30bb78aaf5cb9065b26c867efdf6f96be386a0d4876b0e3dc335cf40bf6c4 WatchSource:0}: Error finding container 4fe30bb78aaf5cb9065b26c867efdf6f96be386a0d4876b0e3dc335cf40bf6c4: Status 404 returned error can't find the container with id 4fe30bb78aaf5cb9065b26c867efdf6f96be386a0d4876b0e3dc335cf40bf6c4 Jul 06 22:34:06.112285 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.112265 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7"] Jul 06 22:34:06.116039 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:34:06.116016 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b4ae13_a3f3_43f3_8f23_e8c7f641d789.slice/crio-cf8700d6e7c3be736a08f07816291ba4e50ce98d5b227b6b10ef454fd6a71eea WatchSource:0}: Error finding container cf8700d6e7c3be736a08f07816291ba4e50ce98d5b227b6b10ef454fd6a71eea: Status 404 returned error can't find the container with id cf8700d6e7c3be736a08f07816291ba4e50ce98d5b227b6b10ef454fd6a71eea Jul 06 22:34:06.215903 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:06.215876 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:06.217660 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:06.217643 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:06.913604 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.913560 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" containerName="manager" containerID="cri-o://851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121" gracePeriod=2 Jul 06 22:34:06.915020 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.914991 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" event={"ID":"b9d038e6-db32-49cd-9483-614d24444708","Type":"ContainerStarted","Data":"78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3"} Jul 06 22:34:06.915144 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.915045 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" event={"ID":"b9d038e6-db32-49cd-9483-614d24444708","Type":"ContainerStarted","Data":"4fe30bb78aaf5cb9065b26c867efdf6f96be386a0d4876b0e3dc335cf40bf6c4"} Jul 06 22:34:06.915144 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.915107 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:06.916309 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:06.916271 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:06.916626 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.916607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" event={"ID":"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789","Type":"ContainerStarted","Data":"5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b"} Jul 06 22:34:06.916693 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.916644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" event={"ID":"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789","Type":"ContainerStarted","Data":"cf8700d6e7c3be736a08f07816291ba4e50ce98d5b227b6b10ef454fd6a71eea"} Jul 06 22:34:06.916738 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.916729 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:06.918417 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:06.918395 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:06.925891 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:06.925862 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdr8g\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdr8g" Jul 06 22:34:06.928361 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:06.928338 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:06.988974 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.988920 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" podStartSLOduration=1.988900783 podStartE2EDuration="1.988900783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:06.987759569 +0000 UTC m=+382.597760778" watchObservedRunningTime="2026-07-06 22:34:06.988900783 +0000 UTC m=+382.598901991" Jul 06 22:34:06.989982 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:06.989950 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" podStartSLOduration=1.989934598 podStartE2EDuration="1.989934598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:06.959595332 +0000 UTC m=+382.569596541" watchObservedRunningTime="2026-07-06 22:34:06.989934598 +0000 UTC m=+382.599935807" Jul 06 22:34:07.003649 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.003619 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10737c7e-4a83-497a-8d3d-3a43748d88b5" path="/var/lib/kubelet/pods/10737c7e-4a83-497a-8d3d-3a43748d88b5/volumes" Jul 06 22:34:07.170557 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.170494 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:07.326043 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.326009 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml4hx\" (UniqueName: \"kubernetes.io/projected/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-kube-api-access-ml4hx\") pod \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\" (UID: \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\") " Jul 06 22:34:07.326270 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.326056 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-extensions-socket-volume\") pod \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\" (UID: \"f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87\") " Jul 06 22:34:07.326381 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.326356 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" (UID: "f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:34:07.328101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.328079 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-kube-api-access-ml4hx" (OuterVolumeSpecName: "kube-api-access-ml4hx") pod "f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" (UID: "f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87"). InnerVolumeSpecName "kube-api-access-ml4hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:34:07.426992 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.426901 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4hx\" (UniqueName: \"kubernetes.io/projected/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-kube-api-access-ml4hx\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:34:07.426992 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.426932 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87-extensions-socket-volume\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:34:07.922761 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.922728 2580 generic.go:356] "Generic (PLEG): container finished" podID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" containerID="851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121" exitCode=2 Jul 06 22:34:07.922925 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.922766 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:07.922925 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.922823 2580 scope.go:117] "RemoveContainer" containerID="851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121" Jul 06 22:34:07.925059 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:07.925038 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:07.931359 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.931343 2580 scope.go:117] "RemoveContainer" containerID="851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121" Jul 06 22:34:07.931594 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:07.931574 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121\": container with ID starting with 851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121 not found: ID does not exist" containerID="851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121" Jul 06 22:34:07.931642 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:07.931602 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121"} err="failed to get container status \"851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121\": rpc error: code = NotFound desc = could not find container \"851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121\": container with ID starting with 851fd3be5980fe5642c639a6abb34019a2ea122d0cf453e25f1b369b7ffc5121 not found: ID does not exist" Jul 06 22:34:07.932835 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:07.932812 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vbhf2\" is forbidden: User \"system:node:ip-10-0-136-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-56.ec2.internal' and this object" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vbhf2" Jul 06 22:34:09.003957 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:09.003928 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87" path="/var/lib/kubelet/pods/f4ab7e06-3ce4-49ce-b7d8-c807c5a69c87/volumes" Jul 06 22:34:17.925044 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:17.925015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:17.925447 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:17.925089 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:17.992459 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:17.992431 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7"] Jul 06 22:34:17.992615 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:17.992594 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" podUID="a9b4ae13-a3f3-43f3-8f23-e8c7f641d789" containerName="manager" containerID="cri-o://5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b" gracePeriod=10 Jul 06 22:34:18.183736 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.183707 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx"] Jul 06 22:34:18.187341 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.187320 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.198780 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.198751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx"] Jul 06 22:34:18.241562 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.241538 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:18.328270 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.328246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l26d5\" (UniqueName: \"kubernetes.io/projected/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-kube-api-access-l26d5\") pod \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\" (UID: \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\") " Jul 06 22:34:18.328445 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.328324 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-extensions-socket-volume\") pod \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\" (UID: \"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789\") " Jul 06 22:34:18.328445 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.328432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hncrq\" (UniqueName: \"kubernetes.io/projected/37e4d454-b160-4df7-8852-62671c00680f-kube-api-access-hncrq\") pod \"kuadrant-operator-controller-manager-86878c769f-r5cmx\" (UID: \"37e4d454-b160-4df7-8852-62671c00680f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.328565 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.328461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e4d454-b160-4df7-8852-62671c00680f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r5cmx\" (UID: \"37e4d454-b160-4df7-8852-62671c00680f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.328715 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.328694 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a9b4ae13-a3f3-43f3-8f23-e8c7f641d789" (UID: "a9b4ae13-a3f3-43f3-8f23-e8c7f641d789"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:34:18.330401 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.330381 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-kube-api-access-l26d5" (OuterVolumeSpecName: "kube-api-access-l26d5") pod "a9b4ae13-a3f3-43f3-8f23-e8c7f641d789" (UID: "a9b4ae13-a3f3-43f3-8f23-e8c7f641d789"). InnerVolumeSpecName "kube-api-access-l26d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:34:18.429806 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.429772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hncrq\" (UniqueName: \"kubernetes.io/projected/37e4d454-b160-4df7-8852-62671c00680f-kube-api-access-hncrq\") pod \"kuadrant-operator-controller-manager-86878c769f-r5cmx\" (UID: \"37e4d454-b160-4df7-8852-62671c00680f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.429970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.429817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e4d454-b160-4df7-8852-62671c00680f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r5cmx\" (UID: \"37e4d454-b160-4df7-8852-62671c00680f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.429970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.429935 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-extensions-socket-volume\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:34:18.429970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.429951 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l26d5\" (UniqueName: \"kubernetes.io/projected/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789-kube-api-access-l26d5\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:34:18.430223 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.430201 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e4d454-b160-4df7-8852-62671c00680f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r5cmx\" (UID: \"37e4d454-b160-4df7-8852-62671c00680f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.438619 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.438547 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hncrq\" (UniqueName: \"kubernetes.io/projected/37e4d454-b160-4df7-8852-62671c00680f-kube-api-access-hncrq\") pod \"kuadrant-operator-controller-manager-86878c769f-r5cmx\" (UID: \"37e4d454-b160-4df7-8852-62671c00680f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.499340 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.499311 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.631947 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.631918 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx"] Jul 06 22:34:18.634370 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:34:18.634344 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e4d454_b160_4df7_8852_62671c00680f.slice/crio-d25f76ec6cef6b78e1eb04fc815506083c294ac87dac3f66c2f561e461a57952 WatchSource:0}: Error finding container d25f76ec6cef6b78e1eb04fc815506083c294ac87dac3f66c2f561e461a57952: Status 404 returned error can't find the container with id d25f76ec6cef6b78e1eb04fc815506083c294ac87dac3f66c2f561e461a57952 Jul 06 22:34:18.978453 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.978422 2580 generic.go:356] "Generic (PLEG): container finished" podID="a9b4ae13-a3f3-43f3-8f23-e8c7f641d789" containerID="5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b" exitCode=0 Jul 06 22:34:18.978844 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.978467 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" Jul 06 22:34:18.978844 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.978510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" event={"ID":"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789","Type":"ContainerDied","Data":"5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b"} Jul 06 22:34:18.978844 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.978560 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7" event={"ID":"a9b4ae13-a3f3-43f3-8f23-e8c7f641d789","Type":"ContainerDied","Data":"cf8700d6e7c3be736a08f07816291ba4e50ce98d5b227b6b10ef454fd6a71eea"} Jul 06 22:34:18.978844 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.978585 2580 scope.go:117] "RemoveContainer" containerID="5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b" Jul 06 22:34:18.980283 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.980255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" event={"ID":"37e4d454-b160-4df7-8852-62671c00680f","Type":"ContainerStarted","Data":"524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c"} Jul 06 22:34:18.980418 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.980297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" event={"ID":"37e4d454-b160-4df7-8852-62671c00680f","Type":"ContainerStarted","Data":"d25f76ec6cef6b78e1eb04fc815506083c294ac87dac3f66c2f561e461a57952"} Jul 06 22:34:18.980418 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.980382 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:18.987352 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.987339 2580 scope.go:117] "RemoveContainer" containerID="5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b" Jul 06 22:34:18.987592 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:18.987576 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b\": container with ID starting with 5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b not found: ID does not exist" containerID="5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b" Jul 06 22:34:18.987632 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:18.987600 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b"} err="failed to get container status \"5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b\": rpc error: code = NotFound desc = could not find container \"5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b\": container with ID starting with 5ebd9c26ba4ef26fe5203a4d0f2380f717992f390f85a431053744d2004ee29b not found: ID does not exist" Jul 06 22:34:19.009012 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:19.008972 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" podStartSLOduration=1.008957838 podStartE2EDuration="1.008957838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:19.008305504 +0000 UTC m=+394.618306716" watchObservedRunningTime="2026-07-06 22:34:19.008957838 +0000 UTC m=+394.618959047" Jul 06 22:34:19.021996 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:19.021966 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7"] Jul 06 22:34:19.034267 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:19.032232 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f6tb7"] Jul 06 22:34:21.003179 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:21.003131 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b4ae13-a3f3-43f3-8f23-e8c7f641d789" path="/var/lib/kubelet/pods/a9b4ae13-a3f3-43f3-8f23-e8c7f641d789/volumes" Jul 06 22:34:29.988595 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:29.988566 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:34:30.035605 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.035563 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb"] Jul 06 22:34:30.035876 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.035843 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" podUID="b9d038e6-db32-49cd-9483-614d24444708" containerName="manager" containerID="cri-o://78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3" gracePeriod=10 Jul 06 22:34:30.300011 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.299984 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:30.335717 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.335692 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d038e6-db32-49cd-9483-614d24444708-extensions-socket-volume\") pod \"b9d038e6-db32-49cd-9483-614d24444708\" (UID: \"b9d038e6-db32-49cd-9483-614d24444708\") " Jul 06 22:34:30.335863 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.335725 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgntd\" (UniqueName: \"kubernetes.io/projected/b9d038e6-db32-49cd-9483-614d24444708-kube-api-access-fgntd\") pod \"b9d038e6-db32-49cd-9483-614d24444708\" (UID: \"b9d038e6-db32-49cd-9483-614d24444708\") " Jul 06 22:34:30.336112 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.336092 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d038e6-db32-49cd-9483-614d24444708-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b9d038e6-db32-49cd-9483-614d24444708" (UID: "b9d038e6-db32-49cd-9483-614d24444708"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:34:30.337733 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.337710 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d038e6-db32-49cd-9483-614d24444708-kube-api-access-fgntd" (OuterVolumeSpecName: "kube-api-access-fgntd") pod "b9d038e6-db32-49cd-9483-614d24444708" (UID: "b9d038e6-db32-49cd-9483-614d24444708"). InnerVolumeSpecName "kube-api-access-fgntd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:34:30.436573 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.436539 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d038e6-db32-49cd-9483-614d24444708-extensions-socket-volume\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:34:30.436573 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:30.436568 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgntd\" (UniqueName: \"kubernetes.io/projected/b9d038e6-db32-49cd-9483-614d24444708-kube-api-access-fgntd\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:34:31.040824 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.040785 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9d038e6-db32-49cd-9483-614d24444708" containerID="78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3" exitCode=0 Jul 06 22:34:31.041229 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.040840 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" Jul 06 22:34:31.041229 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.040901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" event={"ID":"b9d038e6-db32-49cd-9483-614d24444708","Type":"ContainerDied","Data":"78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3"} Jul 06 22:34:31.041229 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.040949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb" event={"ID":"b9d038e6-db32-49cd-9483-614d24444708","Type":"ContainerDied","Data":"4fe30bb78aaf5cb9065b26c867efdf6f96be386a0d4876b0e3dc335cf40bf6c4"} Jul 06 22:34:31.041229 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.040970 2580 scope.go:117] "RemoveContainer" containerID="78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3" Jul 06 22:34:31.049584 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.049567 2580 scope.go:117] "RemoveContainer" containerID="78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3" Jul 06 22:34:31.049861 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:34:31.049843 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3\": container with ID starting with 78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3 not found: ID does not exist" containerID="78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3" Jul 06 22:34:31.049935 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.049868 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3"} err="failed to get container status \"78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3\": rpc error: code = NotFound desc = could not find container \"78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3\": container with ID starting with 78dd082fac59088c64c3418cd8deef0f32952100c6f9acbbf061e5cefae011d3 not found: ID does not exist" Jul 06 22:34:31.060727 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.060699 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb"] Jul 06 22:34:31.063957 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:31.063935 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vwcpb"] Jul 06 22:34:33.003380 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:33.003351 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d038e6-db32-49cd-9483-614d24444708" path="/var/lib/kubelet/pods/b9d038e6-db32-49cd-9483-614d24444708/volumes" Jul 06 22:34:47.389516 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.389473 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ww5hl"] Jul 06 22:34:47.390055 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.390034 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d038e6-db32-49cd-9483-614d24444708" containerName="manager" Jul 06 22:34:47.390137 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.390059 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d038e6-db32-49cd-9483-614d24444708" containerName="manager" Jul 06 22:34:47.390137 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.390081 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b4ae13-a3f3-43f3-8f23-e8c7f641d789" containerName="manager" Jul 06 22:34:47.390137 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.390090 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b4ae13-a3f3-43f3-8f23-e8c7f641d789" containerName="manager" Jul 06 22:34:47.390376 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.390206 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b4ae13-a3f3-43f3-8f23-e8c7f641d789" containerName="manager" Jul 06 22:34:47.390376 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.390226 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d038e6-db32-49cd-9483-614d24444708" containerName="manager" Jul 06 22:34:47.394584 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.394560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:47.397210 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.397188 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pn84x\"" Jul 06 22:34:47.397332 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.397248 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 22:34:47.402970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.402944 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ww5hl"] Jul 06 22:34:47.487348 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.487317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ww5hl"] Jul 06 22:34:47.489563 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.489532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwz6\" (UniqueName: \"kubernetes.io/projected/7a321982-9854-4c33-9951-aefda1a353de-kube-api-access-6gwz6\") pod \"limitador-limitador-64c94fd64f-ww5hl\" (UID: \"7a321982-9854-4c33-9951-aefda1a353de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:47.489686 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.489568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a321982-9854-4c33-9951-aefda1a353de-config-file\") pod \"limitador-limitador-64c94fd64f-ww5hl\" (UID: \"7a321982-9854-4c33-9951-aefda1a353de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:47.590784 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.590753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwz6\" (UniqueName: \"kubernetes.io/projected/7a321982-9854-4c33-9951-aefda1a353de-kube-api-access-6gwz6\") pod \"limitador-limitador-64c94fd64f-ww5hl\" (UID: \"7a321982-9854-4c33-9951-aefda1a353de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:47.590954 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.590791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a321982-9854-4c33-9951-aefda1a353de-config-file\") pod \"limitador-limitador-64c94fd64f-ww5hl\" (UID: \"7a321982-9854-4c33-9951-aefda1a353de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:47.591494 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.591474 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a321982-9854-4c33-9951-aefda1a353de-config-file\") pod \"limitador-limitador-64c94fd64f-ww5hl\" (UID: \"7a321982-9854-4c33-9951-aefda1a353de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:47.598600 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.598566 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwz6\" (UniqueName: \"kubernetes.io/projected/7a321982-9854-4c33-9951-aefda1a353de-kube-api-access-6gwz6\") pod \"limitador-limitador-64c94fd64f-ww5hl\" (UID: \"7a321982-9854-4c33-9951-aefda1a353de\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:47.707018 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.706985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:47.840269 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:47.840241 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ww5hl"] Jul 06 22:34:47.843051 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:34:47.843022 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a321982_9854_4c33_9951_aefda1a353de.slice/crio-3852eb6bb2682541529f7b79f69c1d9c9b72998227be2bcf30da7ac0c972a8ea WatchSource:0}: Error finding container 3852eb6bb2682541529f7b79f69c1d9c9b72998227be2bcf30da7ac0c972a8ea: Status 404 returned error can't find the container with id 3852eb6bb2682541529f7b79f69c1d9c9b72998227be2bcf30da7ac0c972a8ea Jul 06 22:34:48.127412 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.127315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" event={"ID":"7a321982-9854-4c33-9951-aefda1a353de","Type":"ContainerStarted","Data":"3852eb6bb2682541529f7b79f69c1d9c9b72998227be2bcf30da7ac0c972a8ea"} Jul 06 22:34:48.254149 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.254112 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-w8rzl"] Jul 06 22:34:48.258946 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.258924 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-w8rzl" Jul 06 22:34:48.261526 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.261503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lkwzb\"" Jul 06 22:34:48.263683 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.263657 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w8rzl"] Jul 06 22:34:48.297445 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.297410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdr5\" (UniqueName: \"kubernetes.io/projected/58e3e9ba-bb06-444e-9bad-ffa3ab0833a6-kube-api-access-cqdr5\") pod \"authorino-75d556bd77-w8rzl\" (UID: \"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6\") " pod="kuadrant-system/authorino-75d556bd77-w8rzl" Jul 06 22:34:48.398389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.398301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdr5\" (UniqueName: \"kubernetes.io/projected/58e3e9ba-bb06-444e-9bad-ffa3ab0833a6-kube-api-access-cqdr5\") pod \"authorino-75d556bd77-w8rzl\" (UID: \"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6\") " pod="kuadrant-system/authorino-75d556bd77-w8rzl" Jul 06 22:34:48.407112 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.407080 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdr5\" (UniqueName: \"kubernetes.io/projected/58e3e9ba-bb06-444e-9bad-ffa3ab0833a6-kube-api-access-cqdr5\") pod \"authorino-75d556bd77-w8rzl\" (UID: \"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6\") " pod="kuadrant-system/authorino-75d556bd77-w8rzl" Jul 06 22:34:48.570283 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.570208 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-w8rzl" Jul 06 22:34:48.743942 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:48.743913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w8rzl"] Jul 06 22:34:48.747457 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:34:48.747424 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e3e9ba_bb06_444e_9bad_ffa3ab0833a6.slice/crio-10b638fafc8245869aa10a6b01b5d38da5bcde667af32660807e769ac56de9c9 WatchSource:0}: Error finding container 10b638fafc8245869aa10a6b01b5d38da5bcde667af32660807e769ac56de9c9: Status 404 returned error can't find the container with id 10b638fafc8245869aa10a6b01b5d38da5bcde667af32660807e769ac56de9c9 Jul 06 22:34:49.133883 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:49.133799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-w8rzl" event={"ID":"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6","Type":"ContainerStarted","Data":"10b638fafc8245869aa10a6b01b5d38da5bcde667af32660807e769ac56de9c9"} Jul 06 22:34:51.149521 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:51.149477 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" event={"ID":"7a321982-9854-4c33-9951-aefda1a353de","Type":"ContainerStarted","Data":"ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541"} Jul 06 22:34:51.149996 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:51.149556 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:34:51.169330 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:51.169273 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" podStartSLOduration=1.3735650640000001 podStartE2EDuration="4.169252203s" totalImagesPullingTime="2.795687139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:51.166568911 +0000 UTC m=+426.776570150" watchObservedRunningTime="2026-07-06 22:34:51.169252203 +0000 UTC m=+426.779253413" Jul 06 22:34:53.161308 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:53.161274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-w8rzl" event={"ID":"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6","Type":"ContainerStarted","Data":"0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac"} Jul 06 22:34:53.175755 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:34:53.175708 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-w8rzl" podStartSLOduration=1.477347719 podStartE2EDuration="5.17569129s" totalImagesPullingTime="3.698343571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:53.174580492 +0000 UTC m=+428.784581701" watchObservedRunningTime="2026-07-06 22:34:53.17569129 +0000 UTC m=+428.785692498" Jul 06 22:35:02.156845 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:02.156807 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:35:03.875891 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:03.875856 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ww5hl"] Jul 06 22:35:03.876391 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:03.876108 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" podUID="7a321982-9854-4c33-9951-aefda1a353de" containerName="limitador" containerID="cri-o://ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541" gracePeriod=30 Jul 06 22:35:04.428269 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:04.428243 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:35:04.551109 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:04.551009 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gwz6\" (UniqueName: \"kubernetes.io/projected/7a321982-9854-4c33-9951-aefda1a353de-kube-api-access-6gwz6\") pod \"7a321982-9854-4c33-9951-aefda1a353de\" (UID: \"7a321982-9854-4c33-9951-aefda1a353de\") " Jul 06 22:35:04.551109 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:04.551056 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a321982-9854-4c33-9951-aefda1a353de-config-file\") pod \"7a321982-9854-4c33-9951-aefda1a353de\" (UID: \"7a321982-9854-4c33-9951-aefda1a353de\") " Jul 06 22:35:04.551541 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:04.551515 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a321982-9854-4c33-9951-aefda1a353de-config-file" (OuterVolumeSpecName: "config-file") pod "7a321982-9854-4c33-9951-aefda1a353de" (UID: "7a321982-9854-4c33-9951-aefda1a353de"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:35:04.553184 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:04.553142 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a321982-9854-4c33-9951-aefda1a353de-kube-api-access-6gwz6" (OuterVolumeSpecName: "kube-api-access-6gwz6") pod "7a321982-9854-4c33-9951-aefda1a353de" (UID: "7a321982-9854-4c33-9951-aefda1a353de"). InnerVolumeSpecName "kube-api-access-6gwz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:35:04.652410 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:04.652375 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a321982-9854-4c33-9951-aefda1a353de-config-file\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:35:04.652410 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:04.652403 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwz6\" (UniqueName: \"kubernetes.io/projected/7a321982-9854-4c33-9951-aefda1a353de-kube-api-access-6gwz6\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:35:05.223558 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.223515 2580 generic.go:356] "Generic (PLEG): container finished" podID="7a321982-9854-4c33-9951-aefda1a353de" containerID="ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541" exitCode=0 Jul 06 22:35:05.223986 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.223566 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" Jul 06 22:35:05.223986 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.223600 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" event={"ID":"7a321982-9854-4c33-9951-aefda1a353de","Type":"ContainerDied","Data":"ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541"} Jul 06 22:35:05.223986 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.223675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ww5hl" event={"ID":"7a321982-9854-4c33-9951-aefda1a353de","Type":"ContainerDied","Data":"3852eb6bb2682541529f7b79f69c1d9c9b72998227be2bcf30da7ac0c972a8ea"} Jul 06 22:35:05.223986 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.223696 2580 scope.go:117] "RemoveContainer" containerID="ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541" Jul 06 22:35:05.232542 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.232523 2580 scope.go:117] "RemoveContainer" containerID="ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541" Jul 06 22:35:05.232800 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:35:05.232779 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541\": container with ID starting with ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541 not found: ID does not exist" containerID="ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541" Jul 06 22:35:05.232864 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.232812 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541"} err="failed to get container status \"ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541\": rpc error: code = NotFound desc = could not find container \"ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541\": container with ID starting with ba7b8a0dbc50eebf88efad5c52441305e56cb74ec28bd32537a244deeb204541 not found: ID does not exist" Jul 06 22:35:05.240621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.240596 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ww5hl"] Jul 06 22:35:05.244543 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:05.244520 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ww5hl"] Jul 06 22:35:07.003581 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.003485 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a321982-9854-4c33-9951-aefda1a353de" path="/var/lib/kubelet/pods/7a321982-9854-4c33-9951-aefda1a353de/volumes" Jul 06 22:35:07.827663 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.827627 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-t2hnm"] Jul 06 22:35:07.828201 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.828185 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a321982-9854-4c33-9951-aefda1a353de" containerName="limitador" Jul 06 22:35:07.828254 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.828204 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a321982-9854-4c33-9951-aefda1a353de" containerName="limitador" Jul 06 22:35:07.828338 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.828326 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a321982-9854-4c33-9951-aefda1a353de" containerName="limitador" Jul 06 22:35:07.831617 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.831592 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:07.834044 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.834025 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mq9p8\"" Jul 06 22:35:07.834153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.834029 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 22:35:07.842733 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.842710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-t2hnm"] Jul 06 22:35:07.982126 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.982088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8gk\" (UniqueName: \"kubernetes.io/projected/80cdc8e7-ef09-4a48-9179-475df41fb06c-kube-api-access-2s8gk\") pod \"postgres-69fb4594cf-t2hnm\" (UID: \"80cdc8e7-ef09-4a48-9179-475df41fb06c\") " pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:07.982299 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:07.982159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/80cdc8e7-ef09-4a48-9179-475df41fb06c-data\") pod \"postgres-69fb4594cf-t2hnm\" (UID: \"80cdc8e7-ef09-4a48-9179-475df41fb06c\") " pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:08.082924 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:08.082844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/80cdc8e7-ef09-4a48-9179-475df41fb06c-data\") pod \"postgres-69fb4594cf-t2hnm\" (UID: \"80cdc8e7-ef09-4a48-9179-475df41fb06c\") " pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:08.083320 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:08.082926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8gk\" (UniqueName: \"kubernetes.io/projected/80cdc8e7-ef09-4a48-9179-475df41fb06c-kube-api-access-2s8gk\") pod \"postgres-69fb4594cf-t2hnm\" (UID: \"80cdc8e7-ef09-4a48-9179-475df41fb06c\") " pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:08.083320 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:08.083224 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/80cdc8e7-ef09-4a48-9179-475df41fb06c-data\") pod \"postgres-69fb4594cf-t2hnm\" (UID: \"80cdc8e7-ef09-4a48-9179-475df41fb06c\") " pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:08.090843 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:08.090815 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8gk\" (UniqueName: \"kubernetes.io/projected/80cdc8e7-ef09-4a48-9179-475df41fb06c-kube-api-access-2s8gk\") pod \"postgres-69fb4594cf-t2hnm\" (UID: \"80cdc8e7-ef09-4a48-9179-475df41fb06c\") " pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:08.143555 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:08.143519 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:08.273922 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:08.273891 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-t2hnm"] Jul 06 22:35:08.276702 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:35:08.276676 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cdc8e7_ef09_4a48_9179_475df41fb06c.slice/crio-e9c0c6e62653f634ec75305404f507537bfa268b897c564a17b057e197109621 WatchSource:0}: Error finding container e9c0c6e62653f634ec75305404f507537bfa268b897c564a17b057e197109621: Status 404 returned error can't find the container with id e9c0c6e62653f634ec75305404f507537bfa268b897c564a17b057e197109621 Jul 06 22:35:09.248658 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:09.248619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-t2hnm" event={"ID":"80cdc8e7-ef09-4a48-9179-475df41fb06c","Type":"ContainerStarted","Data":"e9c0c6e62653f634ec75305404f507537bfa268b897c564a17b057e197109621"} Jul 06 22:35:13.275468 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:13.275426 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-t2hnm" event={"ID":"80cdc8e7-ef09-4a48-9179-475df41fb06c","Type":"ContainerStarted","Data":"1321614c14f2fa35280248d8b28f39c50560975911503d1a3d93f378ee952988"} Jul 06 22:35:13.275862 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:13.275552 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:19.309086 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:19.309059 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-t2hnm" Jul 06 22:35:19.324693 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:19.324645 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-t2hnm" podStartSLOduration=7.845621698 podStartE2EDuration="12.324626579s" totalImagesPullingTime="4.479004881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:35:13.291404646 +0000 UTC m=+448.901405858" watchObservedRunningTime="2026-07-06 22:35:19.324626579 +0000 UTC m=+454.934627789" Jul 06 22:35:22.023643 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.023612 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w8rzl"] Jul 06 22:35:22.024078 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.023789 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-w8rzl" podUID="58e3e9ba-bb06-444e-9bad-ffa3ab0833a6" containerName="authorino" containerID="cri-o://0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac" gracePeriod=30 Jul 06 22:35:22.313360 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.313290 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-w8rzl" Jul 06 22:35:22.318407 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.318373 2580 generic.go:356] "Generic (PLEG): container finished" podID="58e3e9ba-bb06-444e-9bad-ffa3ab0833a6" containerID="0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac" exitCode=0 Jul 06 22:35:22.318407 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.318408 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-w8rzl" Jul 06 22:35:22.318611 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.318453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-w8rzl" event={"ID":"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6","Type":"ContainerDied","Data":"0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac"} Jul 06 22:35:22.318611 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.318504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-w8rzl" event={"ID":"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6","Type":"ContainerDied","Data":"10b638fafc8245869aa10a6b01b5d38da5bcde667af32660807e769ac56de9c9"} Jul 06 22:35:22.318611 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.318529 2580 scope.go:117] "RemoveContainer" containerID="0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac" Jul 06 22:35:22.326963 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.326941 2580 scope.go:117] "RemoveContainer" containerID="0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac" Jul 06 22:35:22.327358 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:35:22.327330 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac\": container with ID starting with 0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac not found: ID does not exist" containerID="0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac" Jul 06 22:35:22.327454 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.327370 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac"} err="failed to get container status \"0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac\": rpc error: code = NotFound desc = could not find container \"0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac\": container with ID starting with 0999ef586dc0250b76fa715c6cc895a10b69f5d086a0c2c066091f1987e1d3ac not found: ID does not exist" Jul 06 22:35:22.416116 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.416081 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqdr5\" (UniqueName: \"kubernetes.io/projected/58e3e9ba-bb06-444e-9bad-ffa3ab0833a6-kube-api-access-cqdr5\") pod \"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6\" (UID: \"58e3e9ba-bb06-444e-9bad-ffa3ab0833a6\") " Jul 06 22:35:22.418114 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.418089 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e3e9ba-bb06-444e-9bad-ffa3ab0833a6-kube-api-access-cqdr5" (OuterVolumeSpecName: "kube-api-access-cqdr5") pod "58e3e9ba-bb06-444e-9bad-ffa3ab0833a6" (UID: "58e3e9ba-bb06-444e-9bad-ffa3ab0833a6"). InnerVolumeSpecName "kube-api-access-cqdr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:35:22.516723 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.516693 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqdr5\" (UniqueName: \"kubernetes.io/projected/58e3e9ba-bb06-444e-9bad-ffa3ab0833a6-kube-api-access-cqdr5\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:35:22.650333 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.650298 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w8rzl"] Jul 06 22:35:22.652006 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:22.651985 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w8rzl"] Jul 06 22:35:23.003115 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:23.003082 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e3e9ba-bb06-444e-9bad-ffa3ab0833a6" path="/var/lib/kubelet/pods/58e3e9ba-bb06-444e-9bad-ffa3ab0833a6/volumes" Jul 06 22:35:26.376885 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.376851 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5548d895b-8fmcx"] Jul 06 22:35:26.377471 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.377451 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e3e9ba-bb06-444e-9bad-ffa3ab0833a6" containerName="authorino" Jul 06 22:35:26.377471 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.377470 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e3e9ba-bb06-444e-9bad-ffa3ab0833a6" containerName="authorino" Jul 06 22:35:26.377647 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.377620 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e3e9ba-bb06-444e-9bad-ffa3ab0833a6" containerName="authorino" Jul 06 22:35:26.587247 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.587216 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5548d895b-8fmcx"] Jul 06 22:35:26.587430 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.587264 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-gqttz"] Jul 06 22:35:26.587430 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.587332 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:26.591081 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.591044 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 22:35:26.591256 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.591154 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 22:35:26.591256 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.591044 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qslgn\"" Jul 06 22:35:26.612761 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.612739 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-gqttz"] Jul 06 22:35:26.612910 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.612851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:26.651922 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.651837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-webhook-cert\") pod \"maas-controller-5548d895b-8fmcx\" (UID: \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\") " pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:26.651922 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.651886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjfw\" (UniqueName: \"kubernetes.io/projected/0e0f05bd-f299-48be-b28d-0c7827731719-kube-api-access-2gjfw\") pod \"maas-controller-6dcbb958d8-gqttz\" (UID: \"0e0f05bd-f299-48be-b28d-0c7827731719\") " pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:26.652117 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.651939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllvg\" (UniqueName: \"kubernetes.io/projected/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-kube-api-access-sllvg\") pod \"maas-controller-5548d895b-8fmcx\" (UID: \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\") " pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:26.652117 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.651975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e0f05bd-f299-48be-b28d-0c7827731719-webhook-cert\") pod \"maas-controller-6dcbb958d8-gqttz\" (UID: \"0e0f05bd-f299-48be-b28d-0c7827731719\") " pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:26.753354 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.753322 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-webhook-cert\") pod \"maas-controller-5548d895b-8fmcx\" (UID: \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\") " pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:26.753502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.753368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjfw\" (UniqueName: \"kubernetes.io/projected/0e0f05bd-f299-48be-b28d-0c7827731719-kube-api-access-2gjfw\") pod \"maas-controller-6dcbb958d8-gqttz\" (UID: \"0e0f05bd-f299-48be-b28d-0c7827731719\") " pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:26.753502 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.753470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllvg\" (UniqueName: \"kubernetes.io/projected/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-kube-api-access-sllvg\") pod \"maas-controller-5548d895b-8fmcx\" (UID: \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\") " pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:26.753614 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.753503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e0f05bd-f299-48be-b28d-0c7827731719-webhook-cert\") pod \"maas-controller-6dcbb958d8-gqttz\" (UID: \"0e0f05bd-f299-48be-b28d-0c7827731719\") " pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:26.755848 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.755823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e0f05bd-f299-48be-b28d-0c7827731719-webhook-cert\") pod \"maas-controller-6dcbb958d8-gqttz\" (UID: \"0e0f05bd-f299-48be-b28d-0c7827731719\") " pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:26.755960 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.755895 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-webhook-cert\") pod \"maas-controller-5548d895b-8fmcx\" (UID: \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\") " pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:26.761153 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.761128 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllvg\" (UniqueName: \"kubernetes.io/projected/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-kube-api-access-sllvg\") pod \"maas-controller-5548d895b-8fmcx\" (UID: \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\") " pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:26.761408 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.761385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjfw\" (UniqueName: \"kubernetes.io/projected/0e0f05bd-f299-48be-b28d-0c7827731719-kube-api-access-2gjfw\") pod \"maas-controller-6dcbb958d8-gqttz\" (UID: \"0e0f05bd-f299-48be-b28d-0c7827731719\") " pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:26.898477 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.898438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:26.923001 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:26.922985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:27.068965 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:27.068939 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5548d895b-8fmcx"] Jul 06 22:35:27.071398 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:35:27.071368 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8e2a9f_9ec6_4b46_bd71_bcb139bce021.slice/crio-f9f5d602131c7a7c7b31815ee60b449fe9710022789467c7f95d208c0e3599e4 WatchSource:0}: Error finding container f9f5d602131c7a7c7b31815ee60b449fe9710022789467c7f95d208c0e3599e4: Status 404 returned error can't find the container with id f9f5d602131c7a7c7b31815ee60b449fe9710022789467c7f95d208c0e3599e4 Jul 06 22:35:27.095943 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:27.095852 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-gqttz"] Jul 06 22:35:27.343959 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:27.343919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" event={"ID":"0e0f05bd-f299-48be-b28d-0c7827731719","Type":"ContainerStarted","Data":"3e2dbc7613e654bc4c2965de1707179059d22f75a6248c93e133f51a65a6b7bf"} Jul 06 22:35:27.344989 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:27.344966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5548d895b-8fmcx" event={"ID":"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021","Type":"ContainerStarted","Data":"f9f5d602131c7a7c7b31815ee60b449fe9710022789467c7f95d208c0e3599e4"} Jul 06 22:35:29.359829 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:29.359729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5548d895b-8fmcx" event={"ID":"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021","Type":"ContainerStarted","Data":"b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c"} Jul 06 22:35:29.360268 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:29.359895 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:29.361381 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:29.361357 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" event={"ID":"0e0f05bd-f299-48be-b28d-0c7827731719","Type":"ContainerStarted","Data":"ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2"} Jul 06 22:35:29.361461 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:29.361440 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:29.380349 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:29.380304 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5548d895b-8fmcx" podStartSLOduration=1.4819419090000001 podStartE2EDuration="3.380291115s" totalImagesPullingTime="1.898349206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:35:29.378599123 +0000 UTC m=+464.988600365" watchObservedRunningTime="2026-07-06 22:35:29.380291115 +0000 UTC m=+464.990292324" Jul 06 22:35:29.395826 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:29.395782 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" podStartSLOduration=1.515051843 podStartE2EDuration="3.395767321s" totalImagesPullingTime="1.880715478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:35:29.394104156 +0000 UTC m=+465.004105366" watchObservedRunningTime="2026-07-06 22:35:29.395767321 +0000 UTC m=+465.005768572" Jul 06 22:35:40.372605 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.372572 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:40.373086 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.372655 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:35:40.426462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.426424 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5548d895b-8fmcx"] Jul 06 22:35:40.426702 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.426678 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5548d895b-8fmcx" podUID="5f8e2a9f-9ec6-4b46-bd71-bcb139bce021" containerName="manager" containerID="cri-o://b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c" gracePeriod=30 Jul 06 22:35:40.677831 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.677803 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:40.784278 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.784244 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sllvg\" (UniqueName: \"kubernetes.io/projected/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-kube-api-access-sllvg\") pod \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\" (UID: \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\") " Jul 06 22:35:40.784461 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.784323 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-webhook-cert\") pod \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\" (UID: \"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021\") " Jul 06 22:35:40.786450 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.786399 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5f8e2a9f-9ec6-4b46-bd71-bcb139bce021" (UID: "5f8e2a9f-9ec6-4b46-bd71-bcb139bce021"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:35:40.786569 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.786452 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-kube-api-access-sllvg" (OuterVolumeSpecName: "kube-api-access-sllvg") pod "5f8e2a9f-9ec6-4b46-bd71-bcb139bce021" (UID: "5f8e2a9f-9ec6-4b46-bd71-bcb139bce021"). InnerVolumeSpecName "kube-api-access-sllvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:35:40.885691 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.885651 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-webhook-cert\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:35:40.885691 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:40.885684 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sllvg\" (UniqueName: \"kubernetes.io/projected/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021-kube-api-access-sllvg\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:35:41.423370 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.423277 2580 generic.go:356] "Generic (PLEG): container finished" podID="5f8e2a9f-9ec6-4b46-bd71-bcb139bce021" containerID="b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c" exitCode=0 Jul 06 22:35:41.423370 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.423328 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5548d895b-8fmcx" Jul 06 22:35:41.423370 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.423337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5548d895b-8fmcx" event={"ID":"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021","Type":"ContainerDied","Data":"b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c"} Jul 06 22:35:41.423852 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.423390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5548d895b-8fmcx" event={"ID":"5f8e2a9f-9ec6-4b46-bd71-bcb139bce021","Type":"ContainerDied","Data":"f9f5d602131c7a7c7b31815ee60b449fe9710022789467c7f95d208c0e3599e4"} Jul 06 22:35:41.423852 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.423422 2580 scope.go:117] "RemoveContainer" containerID="b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c" Jul 06 22:35:41.432455 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.432277 2580 scope.go:117] "RemoveContainer" containerID="b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c" Jul 06 22:35:41.432564 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:35:41.432546 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c\": container with ID starting with b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c not found: ID does not exist" containerID="b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c" Jul 06 22:35:41.432607 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.432575 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c"} err="failed to get container status \"b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c\": rpc error: code = NotFound desc = could not find container \"b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c\": container with ID starting with b0e9afe2c3ece5ccaff2531ed02f52b5005c2dce8fb6caec4ef55597f3a5684c not found: ID does not exist" Jul 06 22:35:41.438374 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.438350 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5548d895b-8fmcx"] Jul 06 22:35:41.442254 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:41.442228 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5548d895b-8fmcx"] Jul 06 22:35:43.003263 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:35:43.003233 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f8e2a9f-9ec6-4b46-bd71-bcb139bce021" path="/var/lib/kubelet/pods/5f8e2a9f-9ec6-4b46-bd71-bcb139bce021/volumes" Jul 06 22:36:04.678535 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.678495 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-7f647c485-55h88"] Jul 06 22:36:04.679138 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.679116 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8e2a9f-9ec6-4b46-bd71-bcb139bce021" containerName="manager" Jul 06 22:36:04.679229 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.679140 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8e2a9f-9ec6-4b46-bd71-bcb139bce021" containerName="manager" Jul 06 22:36:04.679276 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.679240 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8e2a9f-9ec6-4b46-bd71-bcb139bce021" containerName="manager" Jul 06 22:36:04.686785 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.686747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.689304 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.689258 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 22:36:04.689402 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.689373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-sq7c7\"" Jul 06 22:36:04.693087 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.693060 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-7f647c485-55h88"] Jul 06 22:36:04.695234 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.695212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4741aeb8-144e-43d5-a8db-b483000bf901-openshift-service-ca\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.695326 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.695271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4741aeb8-144e-43d5-a8db-b483000bf901-maas-api-tls\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.695378 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.695326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggq2\" (UniqueName: \"kubernetes.io/projected/4741aeb8-144e-43d5-a8db-b483000bf901-kube-api-access-mggq2\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.796022 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.795989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4741aeb8-144e-43d5-a8db-b483000bf901-openshift-service-ca\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.796261 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.796050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4741aeb8-144e-43d5-a8db-b483000bf901-maas-api-tls\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.796261 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.796081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggq2\" (UniqueName: \"kubernetes.io/projected/4741aeb8-144e-43d5-a8db-b483000bf901-kube-api-access-mggq2\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.796713 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.796686 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4741aeb8-144e-43d5-a8db-b483000bf901-openshift-service-ca\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.798449 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.798426 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4741aeb8-144e-43d5-a8db-b483000bf901-maas-api-tls\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.806029 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.806009 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggq2\" (UniqueName: \"kubernetes.io/projected/4741aeb8-144e-43d5-a8db-b483000bf901-kube-api-access-mggq2\") pod \"maas-api-7f647c485-55h88\" (UID: \"4741aeb8-144e-43d5-a8db-b483000bf901\") " pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:04.998876 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:04.998828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:05.148402 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.148372 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6lhlq"] Jul 06 22:36:05.153674 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.153652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:05.156189 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.156145 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 22:36:05.156481 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.156345 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-f59cp\"" Jul 06 22:36:05.156481 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.156388 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 22:36:05.156821 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.156803 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 22:36:05.163827 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.163246 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6lhlq"] Jul 06 22:36:05.166708 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.166244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-7f647c485-55h88"] Jul 06 22:36:05.166838 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:36:05.166819 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4741aeb8_144e_43d5_a8db_b483000bf901.slice/crio-f4b97708f846ea7c2fd83060b382adc29bdf43fae87e8c52d06703084254908c WatchSource:0}: Error finding container f4b97708f846ea7c2fd83060b382adc29bdf43fae87e8c52d06703084254908c: Status 404 returned error can't find the container with id f4b97708f846ea7c2fd83060b382adc29bdf43fae87e8c52d06703084254908c Jul 06 22:36:05.182923 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.182892 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-w5wll"] Jul 06 22:36:05.187075 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.187051 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:05.196331 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.196304 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-w5wll"] Jul 06 22:36:05.199488 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.199461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fws9f\" (UniqueName: \"kubernetes.io/projected/ea09af4b-fb01-4a7e-b7ab-912c2d218051-kube-api-access-fws9f\") pod \"payload-pre-processing-7964457696-6lhlq\" (UID: \"ea09af4b-fb01-4a7e-b7ab-912c2d218051\") " pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:05.199767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.199502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610bee22-67d7-488f-8c2e-4540e738fbdb-plugins-config-volume\") pod \"payload-processing-58f86c7798-w5wll\" (UID: \"610bee22-67d7-488f-8c2e-4540e738fbdb\") " pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:05.199767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.199574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea09af4b-fb01-4a7e-b7ab-912c2d218051-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6lhlq\" (UID: \"ea09af4b-fb01-4a7e-b7ab-912c2d218051\") " pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:05.199767 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.199641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cm9v\" (UniqueName: \"kubernetes.io/projected/610bee22-67d7-488f-8c2e-4540e738fbdb-kube-api-access-5cm9v\") pod \"payload-processing-58f86c7798-w5wll\" (UID: \"610bee22-67d7-488f-8c2e-4540e738fbdb\") " pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:05.300862 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.300774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fws9f\" (UniqueName: \"kubernetes.io/projected/ea09af4b-fb01-4a7e-b7ab-912c2d218051-kube-api-access-fws9f\") pod \"payload-pre-processing-7964457696-6lhlq\" (UID: \"ea09af4b-fb01-4a7e-b7ab-912c2d218051\") " pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:05.300862 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.300829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610bee22-67d7-488f-8c2e-4540e738fbdb-plugins-config-volume\") pod \"payload-processing-58f86c7798-w5wll\" (UID: \"610bee22-67d7-488f-8c2e-4540e738fbdb\") " pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:05.301070 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.300884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea09af4b-fb01-4a7e-b7ab-912c2d218051-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6lhlq\" (UID: \"ea09af4b-fb01-4a7e-b7ab-912c2d218051\") " pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:05.301070 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.300921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cm9v\" (UniqueName: \"kubernetes.io/projected/610bee22-67d7-488f-8c2e-4540e738fbdb-kube-api-access-5cm9v\") pod \"payload-processing-58f86c7798-w5wll\" (UID: \"610bee22-67d7-488f-8c2e-4540e738fbdb\") " pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:05.301857 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.301833 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610bee22-67d7-488f-8c2e-4540e738fbdb-plugins-config-volume\") pod \"payload-processing-58f86c7798-w5wll\" (UID: \"610bee22-67d7-488f-8c2e-4540e738fbdb\") " pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:05.301898 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.301843 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea09af4b-fb01-4a7e-b7ab-912c2d218051-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6lhlq\" (UID: \"ea09af4b-fb01-4a7e-b7ab-912c2d218051\") " pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:05.309151 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.309123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cm9v\" (UniqueName: \"kubernetes.io/projected/610bee22-67d7-488f-8c2e-4540e738fbdb-kube-api-access-5cm9v\") pod \"payload-processing-58f86c7798-w5wll\" (UID: \"610bee22-67d7-488f-8c2e-4540e738fbdb\") " pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:05.309151 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.309132 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fws9f\" (UniqueName: \"kubernetes.io/projected/ea09af4b-fb01-4a7e-b7ab-912c2d218051-kube-api-access-fws9f\") pod \"payload-pre-processing-7964457696-6lhlq\" (UID: \"ea09af4b-fb01-4a7e-b7ab-912c2d218051\") " pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:05.467426 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.467389 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:05.499323 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.499289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:05.539647 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.539611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-7f647c485-55h88" event={"ID":"4741aeb8-144e-43d5-a8db-b483000bf901","Type":"ContainerStarted","Data":"f4b97708f846ea7c2fd83060b382adc29bdf43fae87e8c52d06703084254908c"} Jul 06 22:36:05.625178 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.625128 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6lhlq"] Jul 06 22:36:05.654879 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:05.654844 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-w5wll"] Jul 06 22:36:05.657993 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:36:05.657959 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610bee22_67d7_488f_8c2e_4540e738fbdb.slice/crio-9d1e85c3c73f30cea8b37076d8b3fd5c1675afcfedff118fb3813167d203e536 WatchSource:0}: Error finding container 9d1e85c3c73f30cea8b37076d8b3fd5c1675afcfedff118fb3813167d203e536: Status 404 returned error can't find the container with id 9d1e85c3c73f30cea8b37076d8b3fd5c1675afcfedff118fb3813167d203e536 Jul 06 22:36:06.548591 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:06.548463 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" event={"ID":"ea09af4b-fb01-4a7e-b7ab-912c2d218051","Type":"ContainerStarted","Data":"7f3bbded01deee0ef2f7a4dbda8a4b9b5081f734f77e9d349533972b03d5fdd2"} Jul 06 22:36:06.552255 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:06.552219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-w5wll" event={"ID":"610bee22-67d7-488f-8c2e-4540e738fbdb","Type":"ContainerStarted","Data":"9d1e85c3c73f30cea8b37076d8b3fd5c1675afcfedff118fb3813167d203e536"} Jul 06 22:36:07.566912 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.566856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-w5wll" event={"ID":"610bee22-67d7-488f-8c2e-4540e738fbdb","Type":"ContainerStarted","Data":"546e3f686df9cc53e8065a721c63411febb3929554a985606b1eb184d79244fc"} Jul 06 22:36:07.567432 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.567121 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:07.568428 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.568394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-7f647c485-55h88" event={"ID":"4741aeb8-144e-43d5-a8db-b483000bf901","Type":"ContainerStarted","Data":"52ac458d5fd052ae26ef871288d4e5efc554acf7ffd4821e748a40538c717fd4"} Jul 06 22:36:07.568565 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.568491 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:07.569804 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.569774 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" event={"ID":"ea09af4b-fb01-4a7e-b7ab-912c2d218051","Type":"ContainerStarted","Data":"af5fc76eff41dd5dbb94b054ff8f1583d94fd74f1f56aec3ce5f471bfd6098fd"} Jul 06 22:36:07.569920 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.569824 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:07.584009 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.583955 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-w5wll" podStartSLOduration=1.209488795 podStartE2EDuration="2.583939188s" totalImagesPullingTime="1.374450393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:36:07.583432119 +0000 UTC m=+503.193433340" watchObservedRunningTime="2026-07-06 22:36:07.583939188 +0000 UTC m=+503.193940401" Jul 06 22:36:07.606874 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.606802 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-7f647c485-55h88" podStartSLOduration=1.7405504 podStartE2EDuration="3.606782747s" totalImagesPullingTime="1.866232347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:36:07.606758087 +0000 UTC m=+503.216759309" watchObservedRunningTime="2026-07-06 22:36:07.606782747 +0000 UTC m=+503.216783957" Jul 06 22:36:07.622478 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:07.622429 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" podStartSLOduration=1.217581721 podStartE2EDuration="2.622413582s" totalImagesPullingTime="1.404831861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:36:07.621227422 +0000 UTC m=+503.231228633" watchObservedRunningTime="2026-07-06 22:36:07.622413582 +0000 UTC m=+503.232414791" Jul 06 22:36:13.581374 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:13.581341 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-7f647c485-55h88" Jul 06 22:36:18.577919 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:18.577890 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-6lhlq" Jul 06 22:36:18.578346 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:18.577970 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-w5wll" Jul 06 22:36:48.593713 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.593679 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf"] Jul 06 22:36:48.600363 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.600340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.603001 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.602814 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 22:36:48.603980 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.603868 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5xqsk\"" Jul 06 22:36:48.603980 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.603931 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 22:36:48.603980 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.603935 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 22:36:48.607610 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.607585 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf"] Jul 06 22:36:48.698064 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.698030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.698265 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.698072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.698265 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.698090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjzp\" (UniqueName: \"kubernetes.io/projected/17c6ddc2-09a8-4561-9510-db903c84ff9a-kube-api-access-rwjzp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.698265 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.698233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c6ddc2-09a8-4561-9510-db903c84ff9a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.698391 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.698274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.698391 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.698297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.798851 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.798816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.798851 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.798848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjzp\" (UniqueName: \"kubernetes.io/projected/17c6ddc2-09a8-4561-9510-db903c84ff9a-kube-api-access-rwjzp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.799084 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.798913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c6ddc2-09a8-4561-9510-db903c84ff9a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.799084 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.798941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.799084 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.798962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.799084 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.798982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.799362 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.799340 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.799416 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.799394 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.799465 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.799447 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.801135 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.801104 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c6ddc2-09a8-4561-9510-db903c84ff9a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.801414 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.801397 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c6ddc2-09a8-4561-9510-db903c84ff9a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.805952 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.805935 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjzp\" (UniqueName: \"kubernetes.io/projected/17c6ddc2-09a8-4561-9510-db903c84ff9a-kube-api-access-rwjzp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf\" (UID: \"17c6ddc2-09a8-4561-9510-db903c84ff9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:48.913983 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:48.913907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:49.057923 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:49.057893 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf"] Jul 06 22:36:49.065682 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:36:49.065653 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c6ddc2_09a8_4561_9510_db903c84ff9a.slice/crio-4632a6dbb33fe2913f7094f4da9703aec6a0d561709f481769396ff8757c2f5c WatchSource:0}: Error finding container 4632a6dbb33fe2913f7094f4da9703aec6a0d561709f481769396ff8757c2f5c: Status 404 returned error can't find the container with id 4632a6dbb33fe2913f7094f4da9703aec6a0d561709f481769396ff8757c2f5c Jul 06 22:36:49.780335 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:49.780295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" event={"ID":"17c6ddc2-09a8-4561-9510-db903c84ff9a","Type":"ContainerStarted","Data":"4632a6dbb33fe2913f7094f4da9703aec6a0d561709f481769396ff8757c2f5c"} Jul 06 22:36:50.790275 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:50.790234 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" event={"ID":"17c6ddc2-09a8-4561-9510-db903c84ff9a","Type":"ContainerStarted","Data":"7998b13ec97ada43d15ca6fad8733a7cb26a6c205fcfd523b6547683037bd38a"} Jul 06 22:36:50.808109 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:50.808058 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" podStartSLOduration=1.798830089 podStartE2EDuration="2.80804067s" totalImagesPullingTime="1.009210581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:36:50.805676568 +0000 UTC m=+546.415677774" watchObservedRunningTime="2026-07-06 22:36:50.80804067 +0000 UTC m=+546.418041879" Jul 06 22:36:58.914658 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:58.914605 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:58.915319 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:58.914683 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:58.927282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:58.927257 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:36:59.846856 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:36:59.846826 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf" Jul 06 22:37:16.488042 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.487989 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx"] Jul 06 22:37:16.491841 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.491821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.494295 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.494275 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 22:37:16.501364 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.501342 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx"] Jul 06 22:37:16.669427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.669398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.669609 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.669435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.669609 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.669467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.669609 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.669485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a69b2a-fd89-4031-9623-3271fef9999f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.669609 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.669531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxc88\" (UniqueName: \"kubernetes.io/projected/30a69b2a-fd89-4031-9623-3271fef9999f-kube-api-access-gxc88\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.669609 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.669586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.770906 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.770814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.770906 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.770849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.770906 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.770877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.770906 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.770896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a69b2a-fd89-4031-9623-3271fef9999f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.771264 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.770924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxc88\" (UniqueName: \"kubernetes.io/projected/30a69b2a-fd89-4031-9623-3271fef9999f-kube-api-access-gxc88\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.771264 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.770984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.771411 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.771392 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.771462 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.771416 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.771497 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.771460 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.773345 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.773315 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a69b2a-fd89-4031-9623-3271fef9999f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.773532 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.773510 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a69b2a-fd89-4031-9623-3271fef9999f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.778250 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.778225 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxc88\" (UniqueName: \"kubernetes.io/projected/30a69b2a-fd89-4031-9623-3271fef9999f-kube-api-access-gxc88\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kmtrx\" (UID: \"30a69b2a-fd89-4031-9623-3271fef9999f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.803219 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.803181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:16.938935 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:16.938912 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx"] Jul 06 22:37:16.941258 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:37:16.941232 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a69b2a_fd89_4031_9623_3271fef9999f.slice/crio-7679d0bc2f85280a62112fd3a69f877e8f94a6e970151790bf8c06ca33c34996 WatchSource:0}: Error finding container 7679d0bc2f85280a62112fd3a69f877e8f94a6e970151790bf8c06ca33c34996: Status 404 returned error can't find the container with id 7679d0bc2f85280a62112fd3a69f877e8f94a6e970151790bf8c06ca33c34996 Jul 06 22:37:17.934533 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:17.934497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" event={"ID":"30a69b2a-fd89-4031-9623-3271fef9999f","Type":"ContainerStarted","Data":"6ce3645c86148b9c66be3aa79703170438b9cb2ad7213de355668cdfe747cced"} Jul 06 22:37:17.934988 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:17.934591 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" event={"ID":"30a69b2a-fd89-4031-9623-3271fef9999f","Type":"ContainerStarted","Data":"7679d0bc2f85280a62112fd3a69f877e8f94a6e970151790bf8c06ca33c34996"} Jul 06 22:37:17.952884 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:17.952837 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" podStartSLOduration=1.750088808 podStartE2EDuration="1.952822401s" totalImagesPullingTime="202.733593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:37:17.950670109 +0000 UTC m=+573.560671330" watchObservedRunningTime="2026-07-06 22:37:17.952822401 +0000 UTC m=+573.562823610" Jul 06 22:37:26.803440 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:26.803400 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:26.803854 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:26.803533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:26.815882 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:26.815857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:27.004049 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:27.004021 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kmtrx" Jul 06 22:37:32.477297 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.477265 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7"] Jul 06 22:37:32.482578 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.482555 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.485002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.484982 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 22:37:32.491525 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.491504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7"] Jul 06 22:37:32.511932 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.511902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.512085 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.511942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.512085 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.512005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.512085 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.512041 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.512085 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.512065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.512314 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.512119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gts4q\" (UniqueName: \"kubernetes.io/projected/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-kube-api-access-gts4q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.613483 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.613443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.613483 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.613483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.613690 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.613532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.613690 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.613571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.613690 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.613609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.613690 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.613654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gts4q\" (UniqueName: \"kubernetes.io/projected/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-kube-api-access-gts4q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.613989 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.613959 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.614086 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.614030 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.614086 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.614063 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.615686 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.615663 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.615926 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.615906 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.621415 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.621399 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gts4q\" (UniqueName: \"kubernetes.io/projected/5b2b906b-8a84-4fb8-b88f-420eaeb4a9da-kube-api-access-gts4q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h29d7\" (UID: \"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.795503 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.795422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:32.928004 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:32.927978 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7"] Jul 06 22:37:32.930225 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:37:32.930199 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2b906b_8a84_4fb8_b88f_420eaeb4a9da.slice/crio-ec5d36e1643257083a514800acbda47d6699177ac98a7e84f0067a814e294f81 WatchSource:0}: Error finding container ec5d36e1643257083a514800acbda47d6699177ac98a7e84f0067a814e294f81: Status 404 returned error can't find the container with id ec5d36e1643257083a514800acbda47d6699177ac98a7e84f0067a814e294f81 Jul 06 22:37:33.016578 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:33.016544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" event={"ID":"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da","Type":"ContainerStarted","Data":"ec5d36e1643257083a514800acbda47d6699177ac98a7e84f0067a814e294f81"} Jul 06 22:37:34.023149 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:34.023117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" event={"ID":"5b2b906b-8a84-4fb8-b88f-420eaeb4a9da","Type":"ContainerStarted","Data":"1cf86cbaf258aa08a0e5f66661b5efbe6f45401e3bb95236465face37f5aea95"} Jul 06 22:37:34.042041 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:34.041951 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" podStartSLOduration=1.873047525 podStartE2EDuration="2.041936495s" totalImagesPullingTime="168.88897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:37:34.040708085 +0000 UTC m=+589.650709295" watchObservedRunningTime="2026-07-06 22:37:34.041936495 +0000 UTC m=+589.651937704" Jul 06 22:37:42.795621 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:42.795585 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:42.796145 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:42.795649 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:42.808042 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:42.808019 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:37:43.083240 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:37:43.083141 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h29d7" Jul 06 22:39:42.916956 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:42.916922 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-gqttz"] Jul 06 22:39:42.917415 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:42.917121 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" podUID="0e0f05bd-f299-48be-b28d-0c7827731719" containerName="manager" containerID="cri-o://ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2" gracePeriod=30 Jul 06 22:39:43.184593 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.184562 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:39:43.295770 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.295745 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gjfw\" (UniqueName: \"kubernetes.io/projected/0e0f05bd-f299-48be-b28d-0c7827731719-kube-api-access-2gjfw\") pod \"0e0f05bd-f299-48be-b28d-0c7827731719\" (UID: \"0e0f05bd-f299-48be-b28d-0c7827731719\") " Jul 06 22:39:43.295942 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.295872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e0f05bd-f299-48be-b28d-0c7827731719-webhook-cert\") pod \"0e0f05bd-f299-48be-b28d-0c7827731719\" (UID: \"0e0f05bd-f299-48be-b28d-0c7827731719\") " Jul 06 22:39:43.297789 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.297756 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0f05bd-f299-48be-b28d-0c7827731719-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0e0f05bd-f299-48be-b28d-0c7827731719" (UID: "0e0f05bd-f299-48be-b28d-0c7827731719"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:39:43.297912 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.297837 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0f05bd-f299-48be-b28d-0c7827731719-kube-api-access-2gjfw" (OuterVolumeSpecName: "kube-api-access-2gjfw") pod "0e0f05bd-f299-48be-b28d-0c7827731719" (UID: "0e0f05bd-f299-48be-b28d-0c7827731719"). InnerVolumeSpecName "kube-api-access-2gjfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:39:43.396552 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.396521 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e0f05bd-f299-48be-b28d-0c7827731719-webhook-cert\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:39:43.396552 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.396549 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjfw\" (UniqueName: \"kubernetes.io/projected/0e0f05bd-f299-48be-b28d-0c7827731719-kube-api-access-2gjfw\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:39:43.684628 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.684591 2580 generic.go:356] "Generic (PLEG): container finished" podID="0e0f05bd-f299-48be-b28d-0c7827731719" containerID="ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2" exitCode=0 Jul 06 22:39:43.684828 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.684637 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" Jul 06 22:39:43.684828 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.684677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" event={"ID":"0e0f05bd-f299-48be-b28d-0c7827731719","Type":"ContainerDied","Data":"ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2"} Jul 06 22:39:43.684828 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.684735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcbb958d8-gqttz" event={"ID":"0e0f05bd-f299-48be-b28d-0c7827731719","Type":"ContainerDied","Data":"3e2dbc7613e654bc4c2965de1707179059d22f75a6248c93e133f51a65a6b7bf"} Jul 06 22:39:43.684828 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.684767 2580 scope.go:117] "RemoveContainer" containerID="ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2" Jul 06 22:39:43.694259 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.694235 2580 scope.go:117] "RemoveContainer" containerID="ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2" Jul 06 22:39:43.694539 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:39:43.694517 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2\": container with ID starting with ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2 not found: ID does not exist" containerID="ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2" Jul 06 22:39:43.694604 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.694550 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2"} err="failed to get container status \"ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2\": rpc error: code = NotFound desc = could not find container \"ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2\": container with ID starting with ed8736bece7099321f18483ff3339d7511b9af720a0ec1588582848b116114f2 not found: ID does not exist" Jul 06 22:39:43.708237 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.708187 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-gqttz"] Jul 06 22:39:43.710427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:43.710404 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-gqttz"] Jul 06 22:39:44.524212 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.524178 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-k9mds"] Jul 06 22:39:44.524633 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.524620 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0f05bd-f299-48be-b28d-0c7827731719" containerName="manager" Jul 06 22:39:44.524633 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.524634 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0f05bd-f299-48be-b28d-0c7827731719" containerName="manager" Jul 06 22:39:44.524718 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.524709 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e0f05bd-f299-48be-b28d-0c7827731719" containerName="manager" Jul 06 22:39:44.528935 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.528915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:44.531826 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.531809 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 22:39:44.532008 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.531991 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 22:39:44.533062 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.533046 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qslgn\"" Jul 06 22:39:44.539753 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.539730 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-k9mds"] Jul 06 22:39:44.708436 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.708397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db5c4bd2-d144-49b1-9fb5-68f01c0de7d3-webhook-cert\") pod \"maas-controller-6dcbb958d8-k9mds\" (UID: \"db5c4bd2-d144-49b1-9fb5-68f01c0de7d3\") " pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:44.708436 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.708442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccd9\" (UniqueName: \"kubernetes.io/projected/db5c4bd2-d144-49b1-9fb5-68f01c0de7d3-kube-api-access-rccd9\") pod \"maas-controller-6dcbb958d8-k9mds\" (UID: \"db5c4bd2-d144-49b1-9fb5-68f01c0de7d3\") " pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:44.809359 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.809267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db5c4bd2-d144-49b1-9fb5-68f01c0de7d3-webhook-cert\") pod \"maas-controller-6dcbb958d8-k9mds\" (UID: \"db5c4bd2-d144-49b1-9fb5-68f01c0de7d3\") " pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:44.809359 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.809307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccd9\" (UniqueName: \"kubernetes.io/projected/db5c4bd2-d144-49b1-9fb5-68f01c0de7d3-kube-api-access-rccd9\") pod \"maas-controller-6dcbb958d8-k9mds\" (UID: \"db5c4bd2-d144-49b1-9fb5-68f01c0de7d3\") " pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:44.813387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.813368 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 22:39:44.819241 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.819221 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccd9\" (UniqueName: \"kubernetes.io/projected/db5c4bd2-d144-49b1-9fb5-68f01c0de7d3-kube-api-access-rccd9\") pod \"maas-controller-6dcbb958d8-k9mds\" (UID: \"db5c4bd2-d144-49b1-9fb5-68f01c0de7d3\") " pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:44.821849 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.821829 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db5c4bd2-d144-49b1-9fb5-68f01c0de7d3-webhook-cert\") pod \"maas-controller-6dcbb958d8-k9mds\" (UID: \"db5c4bd2-d144-49b1-9fb5-68f01c0de7d3\") " pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:44.843809 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.843786 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qslgn\"" Jul 06 22:39:44.851270 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:44.851250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:45.001152 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:39:45.001085 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5c4bd2_d144_49b1_9fb5_68f01c0de7d3.slice/crio-f3012d3a6b4fa47a6596976a9e7241a845158c78802528607b9df75b337742aa WatchSource:0}: Error finding container f3012d3a6b4fa47a6596976a9e7241a845158c78802528607b9df75b337742aa: Status 404 returned error can't find the container with id f3012d3a6b4fa47a6596976a9e7241a845158c78802528607b9df75b337742aa Jul 06 22:39:45.002646 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:45.002622 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:39:45.005598 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:45.005512 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e0f05bd-f299-48be-b28d-0c7827731719" path="/var/lib/kubelet/pods/0e0f05bd-f299-48be-b28d-0c7827731719/volumes" Jul 06 22:39:45.005827 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:45.005812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcbb958d8-k9mds"] Jul 06 22:39:45.588427 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:45.588408 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 22:39:45.696772 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:45.696733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcbb958d8-k9mds" event={"ID":"db5c4bd2-d144-49b1-9fb5-68f01c0de7d3","Type":"ContainerStarted","Data":"57da7c101da6005ffee254cdd24abbb536cd38d082ad50803780f1c9231b2ef7"} Jul 06 22:39:45.696911 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:45.696788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcbb958d8-k9mds" event={"ID":"db5c4bd2-d144-49b1-9fb5-68f01c0de7d3","Type":"ContainerStarted","Data":"f3012d3a6b4fa47a6596976a9e7241a845158c78802528607b9df75b337742aa"} Jul 06 22:39:45.696911 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:45.696813 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:39:45.717549 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:45.717491 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dcbb958d8-k9mds" podStartSLOduration=1.134843684 podStartE2EDuration="1.717471915s" totalImagesPullingTime="582.628231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:39:45.717102486 +0000 UTC m=+721.327103707" watchObservedRunningTime="2026-07-06 22:39:45.717471915 +0000 UTC m=+721.327473123" Jul 06 22:39:56.706362 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:39:56.706326 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dcbb958d8-k9mds" Jul 06 22:44:24.826366 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:24.826286 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx"] Jul 06 22:44:24.828894 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:24.826568 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" podUID="37e4d454-b160-4df7-8852-62671c00680f" containerName="manager" containerID="cri-o://524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c" gracePeriod=10 Jul 06 22:44:27.885848 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:27.885821 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:44:28.020307 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.020220 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hncrq\" (UniqueName: \"kubernetes.io/projected/37e4d454-b160-4df7-8852-62671c00680f-kube-api-access-hncrq\") pod \"37e4d454-b160-4df7-8852-62671c00680f\" (UID: \"37e4d454-b160-4df7-8852-62671c00680f\") " Jul 06 22:44:28.020307 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.020268 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e4d454-b160-4df7-8852-62671c00680f-extensions-socket-volume\") pod \"37e4d454-b160-4df7-8852-62671c00680f\" (UID: \"37e4d454-b160-4df7-8852-62671c00680f\") " Jul 06 22:44:28.020705 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.020679 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e4d454-b160-4df7-8852-62671c00680f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37e4d454-b160-4df7-8852-62671c00680f" (UID: "37e4d454-b160-4df7-8852-62671c00680f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:44:28.022360 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.022336 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e4d454-b160-4df7-8852-62671c00680f-kube-api-access-hncrq" (OuterVolumeSpecName: "kube-api-access-hncrq") pod "37e4d454-b160-4df7-8852-62671c00680f" (UID: "37e4d454-b160-4df7-8852-62671c00680f"). InnerVolumeSpecName "kube-api-access-hncrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:44:28.120959 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.120924 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hncrq\" (UniqueName: \"kubernetes.io/projected/37e4d454-b160-4df7-8852-62671c00680f-kube-api-access-hncrq\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:44:28.120959 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.120953 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e4d454-b160-4df7-8852-62671c00680f-extensions-socket-volume\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:44:28.132449 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.132414 2580 generic.go:356] "Generic (PLEG): container finished" podID="37e4d454-b160-4df7-8852-62671c00680f" containerID="524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c" exitCode=0 Jul 06 22:44:28.132616 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.132458 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" Jul 06 22:44:28.132616 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.132496 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" event={"ID":"37e4d454-b160-4df7-8852-62671c00680f","Type":"ContainerDied","Data":"524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c"} Jul 06 22:44:28.132616 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.132546 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx" event={"ID":"37e4d454-b160-4df7-8852-62671c00680f","Type":"ContainerDied","Data":"d25f76ec6cef6b78e1eb04fc815506083c294ac87dac3f66c2f561e461a57952"} Jul 06 22:44:28.132616 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.132567 2580 scope.go:117] "RemoveContainer" containerID="524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c" Jul 06 22:44:28.143533 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.143514 2580 scope.go:117] "RemoveContainer" containerID="524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c" Jul 06 22:44:28.143831 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:44:28.143811 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c\": container with ID starting with 524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c not found: ID does not exist" containerID="524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c" Jul 06 22:44:28.143902 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.143843 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c"} err="failed to get container status \"524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c\": rpc error: code = NotFound desc = could not find container \"524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c\": container with ID starting with 524880adde32cd7eb707a286ba6b993a64ad4c2fa332530372c6385436003b4c not found: ID does not exist" Jul 06 22:44:28.155500 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.155478 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx"] Jul 06 22:44:28.159007 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:28.158982 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r5cmx"] Jul 06 22:44:29.003823 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:44:29.003791 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e4d454-b160-4df7-8852-62671c00680f" path="/var/lib/kubelet/pods/37e4d454-b160-4df7-8852-62671c00680f/volumes" Jul 06 22:45:00.144367 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.144337 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722965-wkqs5"] Jul 06 22:45:00.144816 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.144745 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e4d454-b160-4df7-8852-62671c00680f" containerName="manager" Jul 06 22:45:00.144816 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.144756 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e4d454-b160-4df7-8852-62671c00680f" containerName="manager" Jul 06 22:45:00.144890 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.144844 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e4d454-b160-4df7-8852-62671c00680f" containerName="manager" Jul 06 22:45:00.152235 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.150127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" Jul 06 22:45:00.156137 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.156109 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722965-wkqs5"] Jul 06 22:45:00.211655 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.211625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq72p\" (UniqueName: \"kubernetes.io/projected/70df8d71-ada9-42b0-95e0-e0a8e0b27c08-kube-api-access-vq72p\") pod \"maas-api-key-cleanup-29722965-wkqs5\" (UID: \"70df8d71-ada9-42b0-95e0-e0a8e0b27c08\") " pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" Jul 06 22:45:00.312594 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.312557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq72p\" (UniqueName: \"kubernetes.io/projected/70df8d71-ada9-42b0-95e0-e0a8e0b27c08-kube-api-access-vq72p\") pod \"maas-api-key-cleanup-29722965-wkqs5\" (UID: \"70df8d71-ada9-42b0-95e0-e0a8e0b27c08\") " pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" Jul 06 22:45:00.321000 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.320971 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq72p\" (UniqueName: \"kubernetes.io/projected/70df8d71-ada9-42b0-95e0-e0a8e0b27c08-kube-api-access-vq72p\") pod \"maas-api-key-cleanup-29722965-wkqs5\" (UID: \"70df8d71-ada9-42b0-95e0-e0a8e0b27c08\") " pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" Jul 06 22:45:00.465728 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.465698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" Jul 06 22:45:00.599175 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.599140 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722965-wkqs5"] Jul 06 22:45:00.600917 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:45:00.600890 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70df8d71_ada9_42b0_95e0_e0a8e0b27c08.slice/crio-949c70b52e3314a03f5367cddd5b8a1a28b7dba66519fad35857fc3cb0c4c3db WatchSource:0}: Error finding container 949c70b52e3314a03f5367cddd5b8a1a28b7dba66519fad35857fc3cb0c4c3db: Status 404 returned error can't find the container with id 949c70b52e3314a03f5367cddd5b8a1a28b7dba66519fad35857fc3cb0c4c3db Jul 06 22:45:00.602787 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:00.602753 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:45:01.301307 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:01.301269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" event={"ID":"70df8d71-ada9-42b0-95e0-e0a8e0b27c08","Type":"ContainerStarted","Data":"949c70b52e3314a03f5367cddd5b8a1a28b7dba66519fad35857fc3cb0c4c3db"} Jul 06 22:45:04.316860 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:04.316832 2580 generic.go:356] "Generic (PLEG): container finished" podID="70df8d71-ada9-42b0-95e0-e0a8e0b27c08" containerID="5e60120106a4352cf01dd26fc51825dfe777a286edb8f1bd89e4881769566266" exitCode=0 Jul 06 22:45:04.317225 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:04.316920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" event={"ID":"70df8d71-ada9-42b0-95e0-e0a8e0b27c08","Type":"ContainerDied","Data":"5e60120106a4352cf01dd26fc51825dfe777a286edb8f1bd89e4881769566266"} Jul 06 22:45:05.456474 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:05.456452 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" Jul 06 22:45:05.564869 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:05.564840 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq72p\" (UniqueName: \"kubernetes.io/projected/70df8d71-ada9-42b0-95e0-e0a8e0b27c08-kube-api-access-vq72p\") pod \"70df8d71-ada9-42b0-95e0-e0a8e0b27c08\" (UID: \"70df8d71-ada9-42b0-95e0-e0a8e0b27c08\") " Jul 06 22:45:05.566890 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:05.566867 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70df8d71-ada9-42b0-95e0-e0a8e0b27c08-kube-api-access-vq72p" (OuterVolumeSpecName: "kube-api-access-vq72p") pod "70df8d71-ada9-42b0-95e0-e0a8e0b27c08" (UID: "70df8d71-ada9-42b0-95e0-e0a8e0b27c08"). InnerVolumeSpecName "kube-api-access-vq72p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:45:05.666186 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:05.666127 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq72p\" (UniqueName: \"kubernetes.io/projected/70df8d71-ada9-42b0-95e0-e0a8e0b27c08-kube-api-access-vq72p\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:45:06.334544 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:06.334514 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" Jul 06 22:45:06.334745 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:06.334513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722965-wkqs5" event={"ID":"70df8d71-ada9-42b0-95e0-e0a8e0b27c08","Type":"ContainerDied","Data":"949c70b52e3314a03f5367cddd5b8a1a28b7dba66519fad35857fc3cb0c4c3db"} Jul 06 22:45:06.334745 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:06.334681 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="949c70b52e3314a03f5367cddd5b8a1a28b7dba66519fad35857fc3cb0c4c3db" Jul 06 22:45:25.983756 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:25.983719 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf"] Jul 06 22:45:25.984192 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:25.984152 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df8d71-ada9-42b0-95e0-e0a8e0b27c08" containerName="cleanup" Jul 06 22:45:25.984192 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:25.984178 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df8d71-ada9-42b0-95e0-e0a8e0b27c08" containerName="cleanup" Jul 06 22:45:25.984276 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:25.984252 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df8d71-ada9-42b0-95e0-e0a8e0b27c08" containerName="cleanup" Jul 06 22:45:25.987330 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:25.987314 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:25.989793 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:25.989765 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 22:45:25.989923 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:25.989865 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 22:45:25.989987 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:25.989953 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jvbzk\"" Jul 06 22:45:26.002326 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.002307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf"] Jul 06 22:45:26.166576 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.166538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49z6\" (UniqueName: \"kubernetes.io/projected/1c546060-facb-41ff-9f3f-edfff60cb4d6-kube-api-access-z49z6\") pod \"kuadrant-operator-controller-manager-86878c769f-vxkkf\" (UID: \"1c546060-facb-41ff-9f3f-edfff60cb4d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:26.166723 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.166647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c546060-facb-41ff-9f3f-edfff60cb4d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vxkkf\" (UID: \"1c546060-facb-41ff-9f3f-edfff60cb4d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:26.267898 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.267813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c546060-facb-41ff-9f3f-edfff60cb4d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vxkkf\" (UID: \"1c546060-facb-41ff-9f3f-edfff60cb4d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:26.268045 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.267903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49z6\" (UniqueName: \"kubernetes.io/projected/1c546060-facb-41ff-9f3f-edfff60cb4d6-kube-api-access-z49z6\") pod \"kuadrant-operator-controller-manager-86878c769f-vxkkf\" (UID: \"1c546060-facb-41ff-9f3f-edfff60cb4d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:26.268221 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.268200 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c546060-facb-41ff-9f3f-edfff60cb4d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vxkkf\" (UID: \"1c546060-facb-41ff-9f3f-edfff60cb4d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:26.280069 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.280032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49z6\" (UniqueName: \"kubernetes.io/projected/1c546060-facb-41ff-9f3f-edfff60cb4d6-kube-api-access-z49z6\") pod \"kuadrant-operator-controller-manager-86878c769f-vxkkf\" (UID: \"1c546060-facb-41ff-9f3f-edfff60cb4d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:26.298305 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.298278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:26.437001 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:26.436969 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf"] Jul 06 22:45:26.440388 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:45:26.440360 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c546060_facb_41ff_9f3f_edfff60cb4d6.slice/crio-07c61d00455bd51d7fd2e570f2f1ad4a23ed5714e0cdd5c3919f927fb511e025 WatchSource:0}: Error finding container 07c61d00455bd51d7fd2e570f2f1ad4a23ed5714e0cdd5c3919f927fb511e025: Status 404 returned error can't find the container with id 07c61d00455bd51d7fd2e570f2f1ad4a23ed5714e0cdd5c3919f927fb511e025 Jul 06 22:45:27.440696 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:27.440664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" event={"ID":"1c546060-facb-41ff-9f3f-edfff60cb4d6","Type":"ContainerStarted","Data":"c6758e3923937cff4b822df93c4466557be79ec65c91e507b5440d1e618bfa02"} Jul 06 22:45:27.441111 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:27.440722 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:45:27.441111 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:27.440739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" event={"ID":"1c546060-facb-41ff-9f3f-edfff60cb4d6","Type":"ContainerStarted","Data":"07c61d00455bd51d7fd2e570f2f1ad4a23ed5714e0cdd5c3919f927fb511e025"} Jul 06 22:45:27.463963 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:27.463915 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" podStartSLOduration=2.463900585 podStartE2EDuration="2.463900585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:45:27.460831227 +0000 UTC m=+1063.070832437" watchObservedRunningTime="2026-07-06 22:45:27.463900585 +0000 UTC m=+1063.073901793" Jul 06 22:45:38.449038 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:45:38.449007 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vxkkf" Jul 06 22:51:43.151801 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.151763 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv"] Jul 06 22:51:43.155967 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.155946 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.158664 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.158639 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-3194e430-serving-cert\"" Jul 06 22:51:43.163910 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.163874 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv"] Jul 06 22:51:43.230906 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.230870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmpl\" (UniqueName: \"kubernetes.io/projected/4bf0481b-5a43-41d2-86ed-64fdd2877e07-kube-api-access-tpmpl\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.231129 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.230927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0481b-5a43-41d2-86ed-64fdd2877e07-openshift-service-ca\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.231228 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.231134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf0481b-5a43-41d2-86ed-64fdd2877e07-maas-api-tls\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.332667 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.332628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf0481b-5a43-41d2-86ed-64fdd2877e07-maas-api-tls\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.332859 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.332723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmpl\" (UniqueName: \"kubernetes.io/projected/4bf0481b-5a43-41d2-86ed-64fdd2877e07-kube-api-access-tpmpl\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.332859 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.332760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0481b-5a43-41d2-86ed-64fdd2877e07-openshift-service-ca\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.333635 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.333607 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0481b-5a43-41d2-86ed-64fdd2877e07-openshift-service-ca\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.335214 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.335188 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf0481b-5a43-41d2-86ed-64fdd2877e07-maas-api-tls\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.340435 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.340414 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmpl\" (UniqueName: \"kubernetes.io/projected/4bf0481b-5a43-41d2-86ed-64fdd2877e07-kube-api-access-tpmpl\") pod \"maas-api-e2e-ait-3194e430-cc68d6749-w9mkv\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.469594 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.469564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:43.616701 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.616674 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv"] Jul 06 22:51:43.619191 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:51:43.619141 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf0481b_5a43_41d2_86ed_64fdd2877e07.slice/crio-6e96bf978136c893e7a4503d9b28b5377ee637b883b9c83c466cce3c2f78a60f WatchSource:0}: Error finding container 6e96bf978136c893e7a4503d9b28b5377ee637b883b9c83c466cce3c2f78a60f: Status 404 returned error can't find the container with id 6e96bf978136c893e7a4503d9b28b5377ee637b883b9c83c466cce3c2f78a60f Jul 06 22:51:43.620540 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:43.620525 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:51:44.393708 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:44.393621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" event={"ID":"4bf0481b-5a43-41d2-86ed-64fdd2877e07","Type":"ContainerStarted","Data":"2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29"} Jul 06 22:51:44.393708 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:44.393679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" event={"ID":"4bf0481b-5a43-41d2-86ed-64fdd2877e07","Type":"ContainerStarted","Data":"6e96bf978136c893e7a4503d9b28b5377ee637b883b9c83c466cce3c2f78a60f"} Jul 06 22:51:44.394113 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:44.393790 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:44.422287 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:44.422240 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" podStartSLOduration=0.96617918 podStartE2EDuration="1.422224966s" totalImagesPullingTime="456.045786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:51:44.419314259 +0000 UTC m=+1440.029315468" watchObservedRunningTime="2026-07-06 22:51:44.422224966 +0000 UTC m=+1440.032226204" Jul 06 22:51:48.554624 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.554590 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv"] Jul 06 22:51:48.555084 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.554893 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" podUID="4bf0481b-5a43-41d2-86ed-64fdd2877e07" containerName="maas-api" containerID="cri-o://2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29" gracePeriod=30 Jul 06 22:51:48.810117 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.810060 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:48.895518 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.895487 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf0481b-5a43-41d2-86ed-64fdd2877e07-maas-api-tls\") pod \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " Jul 06 22:51:48.895676 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.895580 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0481b-5a43-41d2-86ed-64fdd2877e07-openshift-service-ca\") pod \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " Jul 06 22:51:48.895676 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.895619 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpmpl\" (UniqueName: \"kubernetes.io/projected/4bf0481b-5a43-41d2-86ed-64fdd2877e07-kube-api-access-tpmpl\") pod \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\" (UID: \"4bf0481b-5a43-41d2-86ed-64fdd2877e07\") " Jul 06 22:51:48.896007 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.895981 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf0481b-5a43-41d2-86ed-64fdd2877e07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4bf0481b-5a43-41d2-86ed-64fdd2877e07" (UID: "4bf0481b-5a43-41d2-86ed-64fdd2877e07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:51:48.897607 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.897574 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf0481b-5a43-41d2-86ed-64fdd2877e07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4bf0481b-5a43-41d2-86ed-64fdd2877e07" (UID: "4bf0481b-5a43-41d2-86ed-64fdd2877e07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:51:48.897607 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.897586 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf0481b-5a43-41d2-86ed-64fdd2877e07-kube-api-access-tpmpl" (OuterVolumeSpecName: "kube-api-access-tpmpl") pod "4bf0481b-5a43-41d2-86ed-64fdd2877e07" (UID: "4bf0481b-5a43-41d2-86ed-64fdd2877e07"). InnerVolumeSpecName "kube-api-access-tpmpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:51:48.997424 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.997391 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0481b-5a43-41d2-86ed-64fdd2877e07-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:51:48.997424 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.997421 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmpl\" (UniqueName: \"kubernetes.io/projected/4bf0481b-5a43-41d2-86ed-64fdd2877e07-kube-api-access-tpmpl\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:51:48.997424 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:48.997432 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf0481b-5a43-41d2-86ed-64fdd2877e07-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:51:49.422769 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.422734 2580 generic.go:356] "Generic (PLEG): container finished" podID="4bf0481b-5a43-41d2-86ed-64fdd2877e07" containerID="2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29" exitCode=0 Jul 06 22:51:49.422942 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.422789 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" Jul 06 22:51:49.422942 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.422820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" event={"ID":"4bf0481b-5a43-41d2-86ed-64fdd2877e07","Type":"ContainerDied","Data":"2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29"} Jul 06 22:51:49.422942 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.422870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv" event={"ID":"4bf0481b-5a43-41d2-86ed-64fdd2877e07","Type":"ContainerDied","Data":"6e96bf978136c893e7a4503d9b28b5377ee637b883b9c83c466cce3c2f78a60f"} Jul 06 22:51:49.422942 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.422898 2580 scope.go:117] "RemoveContainer" containerID="2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29" Jul 06 22:51:49.432265 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.432250 2580 scope.go:117] "RemoveContainer" containerID="2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29" Jul 06 22:51:49.432525 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:51:49.432508 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29\": container with ID starting with 2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29 not found: ID does not exist" containerID="2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29" Jul 06 22:51:49.432605 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.432531 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29"} err="failed to get container status \"2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29\": rpc error: code = NotFound desc = could not find container \"2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29\": container with ID starting with 2d9cb054a7c4aeb96a1dfdddd5fa31ab37932aed3e48d839653396d6a1a7db29 not found: ID does not exist" Jul 06 22:51:49.443298 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.443274 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv"] Jul 06 22:51:49.445342 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:49.445322 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-3194e430-cc68d6749-w9mkv"] Jul 06 22:51:51.004323 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:51.004295 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bf0481b-5a43-41d2-86ed-64fdd2877e07" path="/var/lib/kubelet/pods/4bf0481b-5a43-41d2-86ed-64fdd2877e07/volumes" Jul 06 22:51:56.330634 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.330587 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk"] Jul 06 22:51:56.331303 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.331285 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf0481b-5a43-41d2-86ed-64fdd2877e07" containerName="maas-api" Jul 06 22:51:56.331303 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.331304 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf0481b-5a43-41d2-86ed-64fdd2877e07" containerName="maas-api" Jul 06 22:51:56.331461 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.331395 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf0481b-5a43-41d2-86ed-64fdd2877e07" containerName="maas-api" Jul 06 22:51:56.336497 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.336471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.339058 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.339031 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-31d40b50-serving-cert\"" Jul 06 22:51:56.346387 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.346362 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk"] Jul 06 22:51:56.368110 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.368080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-openshift-service-ca\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.368282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.368134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.368282 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.368275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74wvn\" (UniqueName: \"kubernetes.io/projected/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-kube-api-access-74wvn\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.469248 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.469214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74wvn\" (UniqueName: \"kubernetes.io/projected/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-kube-api-access-74wvn\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.469430 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.469302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-openshift-service-ca\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.469430 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.469346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.469549 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:51:56.469472 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-31d40b50-serving-cert: secret "maas-api-e2e-ait-31d40b50-serving-cert" not found Jul 06 22:51:56.469601 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:51:56.469554 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls podName:b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e nodeName:}" failed. No retries permitted until 2026-07-06 22:51:56.96953034 +0000 UTC m=+1452.579531532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls") pod "maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" (UID: "b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e") : secret "maas-api-e2e-ait-31d40b50-serving-cert" not found Jul 06 22:51:56.470018 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.469994 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-openshift-service-ca\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.481152 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.481122 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74wvn\" (UniqueName: \"kubernetes.io/projected/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-kube-api-access-74wvn\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.974243 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.974206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:56.976593 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:56.976566 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls\") pod \"maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:57.249366 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:57.249271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:51:57.397708 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:57.397685 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk"] Jul 06 22:51:57.400542 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:51:57.400515 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4822ef8_2b76_4cc3_9d0c_1e3bb0c1868e.slice/crio-e1625f5c0403a3fc1e0a604bb861c99f286dd212eee0750927c4081c2a7cb3b8 WatchSource:0}: Error finding container e1625f5c0403a3fc1e0a604bb861c99f286dd212eee0750927c4081c2a7cb3b8: Status 404 returned error can't find the container with id e1625f5c0403a3fc1e0a604bb861c99f286dd212eee0750927c4081c2a7cb3b8 Jul 06 22:51:57.473231 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:57.473198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" event={"ID":"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e","Type":"ContainerStarted","Data":"e1625f5c0403a3fc1e0a604bb861c99f286dd212eee0750927c4081c2a7cb3b8"} Jul 06 22:51:58.479433 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:58.479396 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" event={"ID":"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e","Type":"ContainerStarted","Data":"a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9"} Jul 06 22:51:58.479797 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:51:58.479463 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:52:01.813377 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:01.813307 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" podStartSLOduration=5.467463105 podStartE2EDuration="5.813284032s" totalImagesPullingTime="345.820927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:51:58.512260355 +0000 UTC m=+1454.122261563" watchObservedRunningTime="2026-07-06 22:52:01.813284032 +0000 UTC m=+1457.423285242" Jul 06 22:52:01.815417 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:01.815391 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk"] Jul 06 22:52:01.815878 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:01.815635 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" podUID="b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" containerName="maas-api" containerID="cri-o://a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9" gracePeriod=30 Jul 06 22:52:02.070385 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.070327 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:52:02.124070 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.124041 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74wvn\" (UniqueName: \"kubernetes.io/projected/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-kube-api-access-74wvn\") pod \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " Jul 06 22:52:02.124218 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.124124 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls\") pod \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " Jul 06 22:52:02.124218 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.124142 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-openshift-service-ca\") pod \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\" (UID: \"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e\") " Jul 06 22:52:02.124555 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.124531 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" (UID: "b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:52:02.126206 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.126182 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" (UID: "b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:52:02.126283 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.126235 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-kube-api-access-74wvn" (OuterVolumeSpecName: "kube-api-access-74wvn") pod "b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" (UID: "b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e"). InnerVolumeSpecName "kube-api-access-74wvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:52:02.225546 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.225513 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:52:02.225546 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.225541 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:52:02.225546 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.225552 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74wvn\" (UniqueName: \"kubernetes.io/projected/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e-kube-api-access-74wvn\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:52:02.502295 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.502264 2580 generic.go:356] "Generic (PLEG): container finished" podID="b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" containerID="a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9" exitCode=0 Jul 06 22:52:02.502450 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.502306 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" Jul 06 22:52:02.502450 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.502344 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" event={"ID":"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e","Type":"ContainerDied","Data":"a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9"} Jul 06 22:52:02.502450 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.502393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk" event={"ID":"b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e","Type":"ContainerDied","Data":"e1625f5c0403a3fc1e0a604bb861c99f286dd212eee0750927c4081c2a7cb3b8"} Jul 06 22:52:02.502450 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.502416 2580 scope.go:117] "RemoveContainer" containerID="a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9" Jul 06 22:52:02.511848 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.511831 2580 scope.go:117] "RemoveContainer" containerID="a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9" Jul 06 22:52:02.512111 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:52:02.512092 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9\": container with ID starting with a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9 not found: ID does not exist" containerID="a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9" Jul 06 22:52:02.512189 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.512120 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9"} err="failed to get container status \"a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9\": rpc error: code = NotFound desc = could not find container \"a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9\": container with ID starting with a99256139896bbb2d8236825d986733caa093ae66c28d4dc3c48b97f5c4fdae9 not found: ID does not exist" Jul 06 22:52:02.539711 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.539685 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk"] Jul 06 22:52:02.546739 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:02.546718 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-31d40b50-5dd58c5945-n9vwk"] Jul 06 22:52:03.005070 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:03.005037 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" path="/var/lib/kubelet/pods/b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e/volumes" Jul 06 22:52:14.874043 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.874014 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks"] Jul 06 22:52:14.874467 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.874455 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" containerName="maas-api" Jul 06 22:52:14.874512 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.874469 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" containerName="maas-api" Jul 06 22:52:14.874550 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.874536 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4822ef8-2b76-4cc3-9d0c-1e3bb0c1868e" containerName="maas-api" Jul 06 22:52:14.877504 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.877489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:14.880648 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.880629 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-aacf6a80-serving-cert\"" Jul 06 22:52:14.898964 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.898939 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks"] Jul 06 22:52:14.950055 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.950018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6th\" (UniqueName: \"kubernetes.io/projected/a08acaac-b169-485b-92d7-b7138d6338fb-kube-api-access-pc6th\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:14.950267 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.950062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08acaac-b169-485b-92d7-b7138d6338fb-maas-api-tls\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:14.950267 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:14.950133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08acaac-b169-485b-92d7-b7138d6338fb-openshift-service-ca\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:15.051682 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.051647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6th\" (UniqueName: \"kubernetes.io/projected/a08acaac-b169-485b-92d7-b7138d6338fb-kube-api-access-pc6th\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:15.051874 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.051710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08acaac-b169-485b-92d7-b7138d6338fb-maas-api-tls\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:15.051874 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.051750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08acaac-b169-485b-92d7-b7138d6338fb-openshift-service-ca\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:15.052453 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.052434 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08acaac-b169-485b-92d7-b7138d6338fb-openshift-service-ca\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:15.054285 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.054263 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08acaac-b169-485b-92d7-b7138d6338fb-maas-api-tls\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:15.061130 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.061105 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6th\" (UniqueName: \"kubernetes.io/projected/a08acaac-b169-485b-92d7-b7138d6338fb-kube-api-access-pc6th\") pod \"maas-api-e2e-derive-aacf6a80-577f866789-6mlks\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:15.187392 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.187356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:15.341873 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.341807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks"] Jul 06 22:52:15.344410 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:52:15.344378 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08acaac_b169_485b_92d7_b7138d6338fb.slice/crio-8af6eba0f8563b84b3dba2a8136c442561cc01169f7ec386acce9e456d79b27f WatchSource:0}: Error finding container 8af6eba0f8563b84b3dba2a8136c442561cc01169f7ec386acce9e456d79b27f: Status 404 returned error can't find the container with id 8af6eba0f8563b84b3dba2a8136c442561cc01169f7ec386acce9e456d79b27f Jul 06 22:52:15.575538 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:15.575447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" event={"ID":"a08acaac-b169-485b-92d7-b7138d6338fb","Type":"ContainerStarted","Data":"8af6eba0f8563b84b3dba2a8136c442561cc01169f7ec386acce9e456d79b27f"} Jul 06 22:52:16.581607 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:16.581568 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" event={"ID":"a08acaac-b169-485b-92d7-b7138d6338fb","Type":"ContainerStarted","Data":"13f8a5667602fc5ae7d75a56100b7c930cd7dd49857e604355a67e5d2613f927"} Jul 06 22:52:16.581993 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:16.581657 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:16.600044 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:16.600000 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" podStartSLOduration=2.077492572 podStartE2EDuration="2.599983936s" totalImagesPullingTime="522.491364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:52:16.598609341 +0000 UTC m=+1472.208610569" watchObservedRunningTime="2026-07-06 22:52:16.599983936 +0000 UTC m=+1472.209985143" Jul 06 22:52:20.463887 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.463844 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks"] Jul 06 22:52:20.464399 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.464022 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" podUID="a08acaac-b169-485b-92d7-b7138d6338fb" containerName="maas-api" containerID="cri-o://13f8a5667602fc5ae7d75a56100b7c930cd7dd49857e604355a67e5d2613f927" gracePeriod=30 Jul 06 22:52:20.470511 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.470487 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:20.604798 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.604764 2580 generic.go:356] "Generic (PLEG): container finished" podID="a08acaac-b169-485b-92d7-b7138d6338fb" containerID="13f8a5667602fc5ae7d75a56100b7c930cd7dd49857e604355a67e5d2613f927" exitCode=0 Jul 06 22:52:20.604970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.604848 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" event={"ID":"a08acaac-b169-485b-92d7-b7138d6338fb","Type":"ContainerDied","Data":"13f8a5667602fc5ae7d75a56100b7c930cd7dd49857e604355a67e5d2613f927"} Jul 06 22:52:20.724016 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.723951 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:20.815246 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.815204 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08acaac-b169-485b-92d7-b7138d6338fb-maas-api-tls\") pod \"a08acaac-b169-485b-92d7-b7138d6338fb\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " Jul 06 22:52:20.815421 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.815317 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08acaac-b169-485b-92d7-b7138d6338fb-openshift-service-ca\") pod \"a08acaac-b169-485b-92d7-b7138d6338fb\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " Jul 06 22:52:20.815421 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.815362 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc6th\" (UniqueName: \"kubernetes.io/projected/a08acaac-b169-485b-92d7-b7138d6338fb-kube-api-access-pc6th\") pod \"a08acaac-b169-485b-92d7-b7138d6338fb\" (UID: \"a08acaac-b169-485b-92d7-b7138d6338fb\") " Jul 06 22:52:20.815672 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.815649 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08acaac-b169-485b-92d7-b7138d6338fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a08acaac-b169-485b-92d7-b7138d6338fb" (UID: "a08acaac-b169-485b-92d7-b7138d6338fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:52:20.817389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.817362 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a08acaac-b169-485b-92d7-b7138d6338fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a08acaac-b169-485b-92d7-b7138d6338fb" (UID: "a08acaac-b169-485b-92d7-b7138d6338fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:52:20.817499 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.817481 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08acaac-b169-485b-92d7-b7138d6338fb-kube-api-access-pc6th" (OuterVolumeSpecName: "kube-api-access-pc6th") pod "a08acaac-b169-485b-92d7-b7138d6338fb" (UID: "a08acaac-b169-485b-92d7-b7138d6338fb"). InnerVolumeSpecName "kube-api-access-pc6th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:52:20.917028 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.916993 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08acaac-b169-485b-92d7-b7138d6338fb-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:52:20.917028 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.917020 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08acaac-b169-485b-92d7-b7138d6338fb-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:52:20.917028 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:20.917030 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6th\" (UniqueName: \"kubernetes.io/projected/a08acaac-b169-485b-92d7-b7138d6338fb-kube-api-access-pc6th\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:52:21.617689 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:21.617657 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" Jul 06 22:52:21.617689 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:21.617669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks" event={"ID":"a08acaac-b169-485b-92d7-b7138d6338fb","Type":"ContainerDied","Data":"8af6eba0f8563b84b3dba2a8136c442561cc01169f7ec386acce9e456d79b27f"} Jul 06 22:52:21.618202 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:21.617746 2580 scope.go:117] "RemoveContainer" containerID="13f8a5667602fc5ae7d75a56100b7c930cd7dd49857e604355a67e5d2613f927" Jul 06 22:52:21.634662 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:21.634634 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks"] Jul 06 22:52:21.639002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:21.638979 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-aacf6a80-577f866789-6mlks"] Jul 06 22:52:23.004083 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:52:23.004049 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a08acaac-b169-485b-92d7-b7138d6338fb" path="/var/lib/kubelet/pods/a08acaac-b169-485b-92d7-b7138d6338fb/volumes" Jul 06 22:54:07.282887 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.282853 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts"] Jul 06 22:54:07.283353 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.283337 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a08acaac-b169-485b-92d7-b7138d6338fb" containerName="maas-api" Jul 06 22:54:07.283400 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.283356 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a08acaac-b169-485b-92d7-b7138d6338fb" containerName="maas-api" Jul 06 22:54:07.283435 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.283422 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a08acaac-b169-485b-92d7-b7138d6338fb" containerName="maas-api" Jul 06 22:54:07.286656 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.286641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:07.288871 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.288849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-b80e969c-serving-cert\"" Jul 06 22:54:07.294571 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.294253 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts"] Jul 06 22:54:07.449187 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.449123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:07.449368 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.449232 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d609e4-9cd4-48ec-8807-dc5c05f708ac-openshift-service-ca\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:07.449368 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.449252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlssw\" (UniqueName: \"kubernetes.io/projected/56d609e4-9cd4-48ec-8807-dc5c05f708ac-kube-api-access-wlssw\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:07.550643 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.550548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:07.550643 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.550599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d609e4-9cd4-48ec-8807-dc5c05f708ac-openshift-service-ca\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:07.550643 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.550616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlssw\" (UniqueName: \"kubernetes.io/projected/56d609e4-9cd4-48ec-8807-dc5c05f708ac-kube-api-access-wlssw\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:07.550901 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:54:07.550725 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-b80e969c-serving-cert: secret "maas-api-e2e-mt-b80e969c-serving-cert" not found Jul 06 22:54:07.550901 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:54:07.550800 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls podName:56d609e4-9cd4-48ec-8807-dc5c05f708ac nodeName:}" failed. No retries permitted until 2026-07-06 22:54:08.050783458 +0000 UTC m=+1583.660784646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls") pod "maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" (UID: "56d609e4-9cd4-48ec-8807-dc5c05f708ac") : secret "maas-api-e2e-mt-b80e969c-serving-cert" not found Jul 06 22:54:07.551240 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.551223 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d609e4-9cd4-48ec-8807-dc5c05f708ac-openshift-service-ca\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:07.558970 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:07.558942 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlssw\" (UniqueName: \"kubernetes.io/projected/56d609e4-9cd4-48ec-8807-dc5c05f708ac-kube-api-access-wlssw\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:08.055512 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:08.055469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:08.055704 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:54:08.055619 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-b80e969c-serving-cert: secret "maas-api-e2e-mt-b80e969c-serving-cert" not found Jul 06 22:54:08.055704 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:54:08.055687 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls podName:56d609e4-9cd4-48ec-8807-dc5c05f708ac nodeName:}" failed. No retries permitted until 2026-07-06 22:54:09.055671348 +0000 UTC m=+1584.665672535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls") pod "maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" (UID: "56d609e4-9cd4-48ec-8807-dc5c05f708ac") : secret "maas-api-e2e-mt-b80e969c-serving-cert" not found Jul 06 22:54:09.065888 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:09.065853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:09.068214 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:09.068191 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls\") pod \"maas-api-e2e-mt-b80e969c-74696f88cf-s6bts\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:09.100010 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:09.099983 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:09.239907 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:09.239877 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts"] Jul 06 22:54:09.242763 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:54:09.242712 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d609e4_9cd4_48ec_8807_dc5c05f708ac.slice/crio-136a71fb3c8a0f499059f72056a551f550218eb8acbfcfe44d0a7ad94c826fd5 WatchSource:0}: Error finding container 136a71fb3c8a0f499059f72056a551f550218eb8acbfcfe44d0a7ad94c826fd5: Status 404 returned error can't find the container with id 136a71fb3c8a0f499059f72056a551f550218eb8acbfcfe44d0a7ad94c826fd5 Jul 06 22:54:10.174948 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:10.174912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" event={"ID":"56d609e4-9cd4-48ec-8807-dc5c05f708ac","Type":"ContainerStarted","Data":"3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1"} Jul 06 22:54:10.175472 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:10.175121 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:10.175472 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:10.175178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" event={"ID":"56d609e4-9cd4-48ec-8807-dc5c05f708ac","Type":"ContainerStarted","Data":"136a71fb3c8a0f499059f72056a551f550218eb8acbfcfe44d0a7ad94c826fd5"} Jul 06 22:54:10.191711 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:10.191660 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" podStartSLOduration=2.777358431 podStartE2EDuration="3.191644363s" totalImagesPullingTime="414.285932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:10.189964363 +0000 UTC m=+1585.799965574" watchObservedRunningTime="2026-07-06 22:54:10.191644363 +0000 UTC m=+1585.801645614" Jul 06 22:54:16.185982 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:16.185956 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:19.523604 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.523569 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts"] Jul 06 22:54:19.524064 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.523819 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" podUID="56d609e4-9cd4-48ec-8807-dc5c05f708ac" containerName="maas-api" containerID="cri-o://3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1" gracePeriod=30 Jul 06 22:54:19.790050 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.789998 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:19.873859 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.873827 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls\") pod \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " Jul 06 22:54:19.874042 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.873891 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlssw\" (UniqueName: \"kubernetes.io/projected/56d609e4-9cd4-48ec-8807-dc5c05f708ac-kube-api-access-wlssw\") pod \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " Jul 06 22:54:19.874042 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.873928 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d609e4-9cd4-48ec-8807-dc5c05f708ac-openshift-service-ca\") pod \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\" (UID: \"56d609e4-9cd4-48ec-8807-dc5c05f708ac\") " Jul 06 22:54:19.874362 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.874337 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d609e4-9cd4-48ec-8807-dc5c05f708ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56d609e4-9cd4-48ec-8807-dc5c05f708ac" (UID: "56d609e4-9cd4-48ec-8807-dc5c05f708ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:54:19.876014 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.875989 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56d609e4-9cd4-48ec-8807-dc5c05f708ac" (UID: "56d609e4-9cd4-48ec-8807-dc5c05f708ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:54:19.876115 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.876044 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d609e4-9cd4-48ec-8807-dc5c05f708ac-kube-api-access-wlssw" (OuterVolumeSpecName: "kube-api-access-wlssw") pod "56d609e4-9cd4-48ec-8807-dc5c05f708ac" (UID: "56d609e4-9cd4-48ec-8807-dc5c05f708ac"). InnerVolumeSpecName "kube-api-access-wlssw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:54:19.974866 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.974828 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlssw\" (UniqueName: \"kubernetes.io/projected/56d609e4-9cd4-48ec-8807-dc5c05f708ac-kube-api-access-wlssw\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:54:19.974866 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.974858 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d609e4-9cd4-48ec-8807-dc5c05f708ac-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:54:19.974866 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:19.974871 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d609e4-9cd4-48ec-8807-dc5c05f708ac-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:54:20.227092 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.227056 2580 generic.go:356] "Generic (PLEG): container finished" podID="56d609e4-9cd4-48ec-8807-dc5c05f708ac" containerID="3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1" exitCode=0 Jul 06 22:54:20.227306 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.227111 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" Jul 06 22:54:20.227306 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.227150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" event={"ID":"56d609e4-9cd4-48ec-8807-dc5c05f708ac","Type":"ContainerDied","Data":"3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1"} Jul 06 22:54:20.227306 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.227235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts" event={"ID":"56d609e4-9cd4-48ec-8807-dc5c05f708ac","Type":"ContainerDied","Data":"136a71fb3c8a0f499059f72056a551f550218eb8acbfcfe44d0a7ad94c826fd5"} Jul 06 22:54:20.227306 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.227263 2580 scope.go:117] "RemoveContainer" containerID="3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1" Jul 06 22:54:20.238954 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.238934 2580 scope.go:117] "RemoveContainer" containerID="3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1" Jul 06 22:54:20.239305 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:54:20.239274 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1\": container with ID starting with 3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1 not found: ID does not exist" containerID="3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1" Jul 06 22:54:20.239467 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.239315 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1"} err="failed to get container status \"3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1\": rpc error: code = NotFound desc = could not find container \"3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1\": container with ID starting with 3937918c43c9cf83061879046f5877a8bb5d3a796271c9755568d8ce36e140b1 not found: ID does not exist" Jul 06 22:54:20.251343 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.251318 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts"] Jul 06 22:54:20.253293 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:20.253271 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-b80e969c-74696f88cf-s6bts"] Jul 06 22:54:21.004375 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:21.004345 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d609e4-9cd4-48ec-8807-dc5c05f708ac" path="/var/lib/kubelet/pods/56d609e4-9cd4-48ec-8807-dc5c05f708ac/volumes" Jul 06 22:54:50.560532 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.560503 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk"] Jul 06 22:54:50.560967 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.560931 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d609e4-9cd4-48ec-8807-dc5c05f708ac" containerName="maas-api" Jul 06 22:54:50.560967 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.560941 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d609e4-9cd4-48ec-8807-dc5c05f708ac" containerName="maas-api" Jul 06 22:54:50.561049 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.561038 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d609e4-9cd4-48ec-8807-dc5c05f708ac" containerName="maas-api" Jul 06 22:54:50.564233 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.564218 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.567671 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.567649 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-2513e2ce-serving-cert\"" Jul 06 22:54:50.589005 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.588975 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk"] Jul 06 22:54:50.652410 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.652370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c50c0f-d7d4-4d8c-8374-804823d7f586-maas-api-tls\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.652601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.652449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c50c0f-d7d4-4d8c-8374-804823d7f586-openshift-service-ca\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.652601 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.652489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wfp\" (UniqueName: \"kubernetes.io/projected/79c50c0f-d7d4-4d8c-8374-804823d7f586-kube-api-access-x5wfp\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.753902 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.753861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c50c0f-d7d4-4d8c-8374-804823d7f586-openshift-service-ca\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.753902 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.753907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wfp\" (UniqueName: \"kubernetes.io/projected/79c50c0f-d7d4-4d8c-8374-804823d7f586-kube-api-access-x5wfp\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.754206 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.753995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c50c0f-d7d4-4d8c-8374-804823d7f586-maas-api-tls\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.754550 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.754526 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c50c0f-d7d4-4d8c-8374-804823d7f586-openshift-service-ca\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.756664 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.756637 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c50c0f-d7d4-4d8c-8374-804823d7f586-maas-api-tls\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.762717 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.762699 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wfp\" (UniqueName: \"kubernetes.io/projected/79c50c0f-d7d4-4d8c-8374-804823d7f586-kube-api-access-x5wfp\") pod \"maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:50.875114 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:50.875021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:51.015487 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:51.015466 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk"] Jul 06 22:54:51.017768 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:54:51.017730 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c50c0f_d7d4_4d8c_8374_804823d7f586.slice/crio-407db0b3b388bf9a68f6dbf920d6b5c0d38e5c7f8efe2eaba7d538fa4f5223e8 WatchSource:0}: Error finding container 407db0b3b388bf9a68f6dbf920d6b5c0d38e5c7f8efe2eaba7d538fa4f5223e8: Status 404 returned error can't find the container with id 407db0b3b388bf9a68f6dbf920d6b5c0d38e5c7f8efe2eaba7d538fa4f5223e8 Jul 06 22:54:51.390033 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:51.390002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" event={"ID":"79c50c0f-d7d4-4d8c-8374-804823d7f586","Type":"ContainerStarted","Data":"407db0b3b388bf9a68f6dbf920d6b5c0d38e5c7f8efe2eaba7d538fa4f5223e8"} Jul 06 22:54:52.397328 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:52.397240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" event={"ID":"79c50c0f-d7d4-4d8c-8374-804823d7f586","Type":"ContainerStarted","Data":"f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a"} Jul 06 22:54:52.397705 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:52.397332 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:54:52.420002 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:52.419939 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" podStartSLOduration=2.108989752 podStartE2EDuration="2.419917186s" totalImagesPullingTime="310.927434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:52.418940286 +0000 UTC m=+1628.028941515" watchObservedRunningTime="2026-07-06 22:54:52.419917186 +0000 UTC m=+1628.029918396" Jul 06 22:54:58.408022 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:54:58.407992 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:55:32.770755 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:32.770709 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk"] Jul 06 22:55:32.771389 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:32.770975 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" podUID="79c50c0f-d7d4-4d8c-8374-804823d7f586" containerName="maas-api" containerID="cri-o://f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a" gracePeriod=30 Jul 06 22:55:33.027768 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.027702 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:55:33.161894 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.161855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c50c0f-d7d4-4d8c-8374-804823d7f586-openshift-service-ca\") pod \"79c50c0f-d7d4-4d8c-8374-804823d7f586\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " Jul 06 22:55:33.162075 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.162051 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c50c0f-d7d4-4d8c-8374-804823d7f586-maas-api-tls\") pod \"79c50c0f-d7d4-4d8c-8374-804823d7f586\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " Jul 06 22:55:33.162138 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.162120 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5wfp\" (UniqueName: \"kubernetes.io/projected/79c50c0f-d7d4-4d8c-8374-804823d7f586-kube-api-access-x5wfp\") pod \"79c50c0f-d7d4-4d8c-8374-804823d7f586\" (UID: \"79c50c0f-d7d4-4d8c-8374-804823d7f586\") " Jul 06 22:55:33.162315 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.162289 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c50c0f-d7d4-4d8c-8374-804823d7f586-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79c50c0f-d7d4-4d8c-8374-804823d7f586" (UID: "79c50c0f-d7d4-4d8c-8374-804823d7f586"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:55:33.162485 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.162470 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c50c0f-d7d4-4d8c-8374-804823d7f586-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:55:33.164148 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.164123 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c50c0f-d7d4-4d8c-8374-804823d7f586-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79c50c0f-d7d4-4d8c-8374-804823d7f586" (UID: "79c50c0f-d7d4-4d8c-8374-804823d7f586"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:55:33.164328 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.164308 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c50c0f-d7d4-4d8c-8374-804823d7f586-kube-api-access-x5wfp" (OuterVolumeSpecName: "kube-api-access-x5wfp") pod "79c50c0f-d7d4-4d8c-8374-804823d7f586" (UID: "79c50c0f-d7d4-4d8c-8374-804823d7f586"). InnerVolumeSpecName "kube-api-access-x5wfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:55:33.263101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.263064 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c50c0f-d7d4-4d8c-8374-804823d7f586-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:55:33.263101 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.263094 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5wfp\" (UniqueName: \"kubernetes.io/projected/79c50c0f-d7d4-4d8c-8374-804823d7f586-kube-api-access-x5wfp\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:55:33.605400 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.605361 2580 generic.go:356] "Generic (PLEG): container finished" podID="79c50c0f-d7d4-4d8c-8374-804823d7f586" containerID="f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a" exitCode=0 Jul 06 22:55:33.605615 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.605427 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" Jul 06 22:55:33.605615 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.605443 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" event={"ID":"79c50c0f-d7d4-4d8c-8374-804823d7f586","Type":"ContainerDied","Data":"f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a"} Jul 06 22:55:33.605615 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.605531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk" event={"ID":"79c50c0f-d7d4-4d8c-8374-804823d7f586","Type":"ContainerDied","Data":"407db0b3b388bf9a68f6dbf920d6b5c0d38e5c7f8efe2eaba7d538fa4f5223e8"} Jul 06 22:55:33.605615 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.605553 2580 scope.go:117] "RemoveContainer" containerID="f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a" Jul 06 22:55:33.614989 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.614972 2580 scope.go:117] "RemoveContainer" containerID="f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a" Jul 06 22:55:33.615313 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:55:33.615293 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a\": container with ID starting with f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a not found: ID does not exist" containerID="f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a" Jul 06 22:55:33.615365 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.615323 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a"} err="failed to get container status \"f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a\": rpc error: code = NotFound desc = could not find container \"f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a\": container with ID starting with f526468ea00c28e5d11e99cc29e2e196c558c82a79fdd10f6e2188b2f548fe7a not found: ID does not exist" Jul 06 22:55:33.626292 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.626268 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk"] Jul 06 22:55:33.629244 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:33.629223 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-2513e2ce-56b5b46dff-8xhhk"] Jul 06 22:55:35.004278 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:55:35.004248 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c50c0f-d7d4-4d8c-8374-804823d7f586" path="/var/lib/kubelet/pods/79c50c0f-d7d4-4d8c-8374-804823d7f586/volumes" Jul 06 22:57:30.191645 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.191611 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s"] Jul 06 22:57:30.192529 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.192426 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c50c0f-d7d4-4d8c-8374-804823d7f586" containerName="maas-api" Jul 06 22:57:30.192529 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.192452 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c50c0f-d7d4-4d8c-8374-804823d7f586" containerName="maas-api" Jul 06 22:57:30.192722 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.192566 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c50c0f-d7d4-4d8c-8374-804823d7f586" containerName="maas-api" Jul 06 22:57:30.195876 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.195855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.198308 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.198289 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-6d2b52-serving-cert\"" Jul 06 22:57:30.212082 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.212052 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s"] Jul 06 22:57:30.287138 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.287102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.287333 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.287196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.287333 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.287277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9t9\" (UniqueName: \"kubernetes.io/projected/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-kube-api-access-5h9t9\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.388264 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.388232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.388445 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.388286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.388445 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.388342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9t9\" (UniqueName: \"kubernetes.io/projected/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-kube-api-access-5h9t9\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.388957 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.388932 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.390659 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.390636 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.395919 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.395896 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9t9\" (UniqueName: \"kubernetes.io/projected/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-kube-api-access-5h9t9\") pod \"maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.510080 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.509999 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:30.658194 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.658123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s"] Jul 06 22:57:30.660778 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:57:30.660746 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99caa7c9_fb85_4f94_be50_b0f8b4b8acc1.slice/crio-fa7b0cc9c19a6a95ced864f06fb29bf7f88f9d266f47eeef2cc313fc9ae73ea2 WatchSource:0}: Error finding container fa7b0cc9c19a6a95ced864f06fb29bf7f88f9d266f47eeef2cc313fc9ae73ea2: Status 404 returned error can't find the container with id fa7b0cc9c19a6a95ced864f06fb29bf7f88f9d266f47eeef2cc313fc9ae73ea2 Jul 06 22:57:30.662072 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:30.662057 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:57:31.227279 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:31.227247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" event={"ID":"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1","Type":"ContainerStarted","Data":"6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c"} Jul 06 22:57:31.227634 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:31.227298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" event={"ID":"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1","Type":"ContainerStarted","Data":"fa7b0cc9c19a6a95ced864f06fb29bf7f88f9d266f47eeef2cc313fc9ae73ea2"} Jul 06 22:57:31.227634 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:31.227388 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:31.268533 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:31.268484 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" podStartSLOduration=0.810644446 podStartE2EDuration="1.268467084s" totalImagesPullingTime="457.822638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:31.262560087 +0000 UTC m=+1786.872561309" watchObservedRunningTime="2026-07-06 22:57:31.268467084 +0000 UTC m=+1786.878468293" Jul 06 22:57:37.239190 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:37.239091 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:57:38.068549 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.068522 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp"] Jul 06 22:57:38.073357 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.073336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.076293 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.076265 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-63da37-serving-cert\"" Jul 06 22:57:38.082368 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.082347 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp"] Jul 06 22:57:38.270041 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.270011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqwr\" (UniqueName: \"kubernetes.io/projected/9d989c08-9187-4e74-ba47-0dab96b7e113-kube-api-access-gkqwr\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.270445 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.270054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d989c08-9187-4e74-ba47-0dab96b7e113-maas-api-tls\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.270445 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.270089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d989c08-9187-4e74-ba47-0dab96b7e113-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.371141 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.371055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqwr\" (UniqueName: \"kubernetes.io/projected/9d989c08-9187-4e74-ba47-0dab96b7e113-kube-api-access-gkqwr\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.371141 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.371097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d989c08-9187-4e74-ba47-0dab96b7e113-maas-api-tls\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.371141 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.371136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d989c08-9187-4e74-ba47-0dab96b7e113-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.371862 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.371837 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d989c08-9187-4e74-ba47-0dab96b7e113-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.373486 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.373467 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d989c08-9187-4e74-ba47-0dab96b7e113-maas-api-tls\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.379388 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.379363 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqwr\" (UniqueName: \"kubernetes.io/projected/9d989c08-9187-4e74-ba47-0dab96b7e113-kube-api-access-gkqwr\") pod \"maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.386682 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.386646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:38.535758 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:38.535725 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp"] Jul 06 22:57:38.536859 ip-10-0-136-56 kubenswrapper[2580]: W0706 22:57:38.536829 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d989c08_9187_4e74_ba47_0dab96b7e113.slice/crio-ad294366df7f5fdcdb7fe2f4b168f475aada3880e0315b75577421c8e9985d38 WatchSource:0}: Error finding container ad294366df7f5fdcdb7fe2f4b168f475aada3880e0315b75577421c8e9985d38: Status 404 returned error can't find the container with id ad294366df7f5fdcdb7fe2f4b168f475aada3880e0315b75577421c8e9985d38 Jul 06 22:57:39.275295 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:39.275249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" event={"ID":"9d989c08-9187-4e74-ba47-0dab96b7e113","Type":"ContainerStarted","Data":"90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c"} Jul 06 22:57:39.275666 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:39.275317 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:57:39.275666 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:39.275341 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" event={"ID":"9d989c08-9187-4e74-ba47-0dab96b7e113","Type":"ContainerStarted","Data":"ad294366df7f5fdcdb7fe2f4b168f475aada3880e0315b75577421c8e9985d38"} Jul 06 22:57:39.295377 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:39.295321 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" podStartSLOduration=0.954867458 podStartE2EDuration="1.295305493s" totalImagesPullingTime="340.438035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:39.292859914 +0000 UTC m=+1794.902861124" watchObservedRunningTime="2026-07-06 22:57:39.295305493 +0000 UTC m=+1794.905306702" Jul 06 22:57:45.285100 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:57:45.285069 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 22:59:24.188215 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.188099 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s"] Jul 06 22:59:24.188799 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.188429 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" podUID="99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" containerName="maas-api" containerID="cri-o://6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c" gracePeriod=30 Jul 06 22:59:24.446590 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.446528 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:59:24.625037 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.625000 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-maas-api-tls\") pod \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " Jul 06 22:59:24.625037 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.625045 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h9t9\" (UniqueName: \"kubernetes.io/projected/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-kube-api-access-5h9t9\") pod \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " Jul 06 22:59:24.625335 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.625209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-openshift-service-ca\") pod \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\" (UID: \"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1\") " Jul 06 22:59:24.625539 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.625505 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" (UID: "99caa7c9-fb85-4f94-be50-b0f8b4b8acc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:59:24.627174 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.627140 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-kube-api-access-5h9t9" (OuterVolumeSpecName: "kube-api-access-5h9t9") pod "99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" (UID: "99caa7c9-fb85-4f94-be50-b0f8b4b8acc1"). InnerVolumeSpecName "kube-api-access-5h9t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:59:24.627256 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.627176 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" (UID: "99caa7c9-fb85-4f94-be50-b0f8b4b8acc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:59:24.726770 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.726684 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:59:24.726770 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.726714 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:59:24.726770 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.726723 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h9t9\" (UniqueName: \"kubernetes.io/projected/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1-kube-api-access-5h9t9\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 22:59:24.836493 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.836462 2580 generic.go:356] "Generic (PLEG): container finished" podID="99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" containerID="6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c" exitCode=0 Jul 06 22:59:24.836689 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.836511 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" Jul 06 22:59:24.836689 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.836557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" event={"ID":"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1","Type":"ContainerDied","Data":"6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c"} Jul 06 22:59:24.836689 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.836618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s" event={"ID":"99caa7c9-fb85-4f94-be50-b0f8b4b8acc1","Type":"ContainerDied","Data":"fa7b0cc9c19a6a95ced864f06fb29bf7f88f9d266f47eeef2cc313fc9ae73ea2"} Jul 06 22:59:24.836689 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.836649 2580 scope.go:117] "RemoveContainer" containerID="6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c" Jul 06 22:59:24.846663 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.846648 2580 scope.go:117] "RemoveContainer" containerID="6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c" Jul 06 22:59:24.846945 ip-10-0-136-56 kubenswrapper[2580]: E0706 22:59:24.846926 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c\": container with ID starting with 6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c not found: ID does not exist" containerID="6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c" Jul 06 22:59:24.847005 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.846957 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c"} err="failed to get container status \"6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c\": rpc error: code = NotFound desc = could not find container \"6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c\": container with ID starting with 6fddd336a35d1124478d2484f819954264f30d38db66a6737b80a89c9ef6588c not found: ID does not exist" Jul 06 22:59:24.859210 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.859184 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s"] Jul 06 22:59:24.862737 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:24.862717 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6d2b52-86cf88fd44-4c78s"] Jul 06 22:59:25.004176 ip-10-0-136-56 kubenswrapper[2580]: I0706 22:59:25.004079 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" path="/var/lib/kubelet/pods/99caa7c9-fb85-4f94-be50-b0f8b4b8acc1/volumes" Jul 06 23:00:07.131399 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.131362 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp"] Jul 06 23:00:07.131950 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.131631 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" podUID="9d989c08-9187-4e74-ba47-0dab96b7e113" containerName="maas-api" containerID="cri-o://90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c" gracePeriod=30 Jul 06 23:00:07.393091 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.393023 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 23:00:07.522524 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.522484 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d989c08-9187-4e74-ba47-0dab96b7e113-maas-api-tls\") pod \"9d989c08-9187-4e74-ba47-0dab96b7e113\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " Jul 06 23:00:07.522710 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.522565 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkqwr\" (UniqueName: \"kubernetes.io/projected/9d989c08-9187-4e74-ba47-0dab96b7e113-kube-api-access-gkqwr\") pod \"9d989c08-9187-4e74-ba47-0dab96b7e113\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " Jul 06 23:00:07.522755 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.522703 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d989c08-9187-4e74-ba47-0dab96b7e113-openshift-service-ca\") pod \"9d989c08-9187-4e74-ba47-0dab96b7e113\" (UID: \"9d989c08-9187-4e74-ba47-0dab96b7e113\") " Jul 06 23:00:07.523037 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.523012 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d989c08-9187-4e74-ba47-0dab96b7e113-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d989c08-9187-4e74-ba47-0dab96b7e113" (UID: "9d989c08-9187-4e74-ba47-0dab96b7e113"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:00:07.523187 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.523140 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d989c08-9187-4e74-ba47-0dab96b7e113-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:07.524563 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.524542 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d989c08-9187-4e74-ba47-0dab96b7e113-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d989c08-9187-4e74-ba47-0dab96b7e113" (UID: "9d989c08-9187-4e74-ba47-0dab96b7e113"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:00:07.524646 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.524585 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d989c08-9187-4e74-ba47-0dab96b7e113-kube-api-access-gkqwr" (OuterVolumeSpecName: "kube-api-access-gkqwr") pod "9d989c08-9187-4e74-ba47-0dab96b7e113" (UID: "9d989c08-9187-4e74-ba47-0dab96b7e113"). InnerVolumeSpecName "kube-api-access-gkqwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:07.624354 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.624320 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqwr\" (UniqueName: \"kubernetes.io/projected/9d989c08-9187-4e74-ba47-0dab96b7e113-kube-api-access-gkqwr\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:07.624354 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:07.624349 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d989c08-9187-4e74-ba47-0dab96b7e113-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:08.058551 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.058517 2580 generic.go:356] "Generic (PLEG): container finished" podID="9d989c08-9187-4e74-ba47-0dab96b7e113" containerID="90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c" exitCode=0 Jul 06 23:00:08.058723 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.058562 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" Jul 06 23:00:08.058723 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.058607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" event={"ID":"9d989c08-9187-4e74-ba47-0dab96b7e113","Type":"ContainerDied","Data":"90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c"} Jul 06 23:00:08.058723 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.058659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp" event={"ID":"9d989c08-9187-4e74-ba47-0dab96b7e113","Type":"ContainerDied","Data":"ad294366df7f5fdcdb7fe2f4b168f475aada3880e0315b75577421c8e9985d38"} Jul 06 23:00:08.058723 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.058681 2580 scope.go:117] "RemoveContainer" containerID="90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c" Jul 06 23:00:08.068384 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.068365 2580 scope.go:117] "RemoveContainer" containerID="90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c" Jul 06 23:00:08.068654 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:00:08.068634 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c\": container with ID starting with 90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c not found: ID does not exist" containerID="90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c" Jul 06 23:00:08.068713 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.068662 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c"} err="failed to get container status \"90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c\": rpc error: code = NotFound desc = could not find container \"90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c\": container with ID starting with 90d622dff8cee0b812ca37bfd9b0daea0e4211cd3d90fcd8e891f4702374ae5c not found: ID does not exist" Jul 06 23:00:08.081026 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.080997 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp"] Jul 06 23:00:08.083080 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:08.083060 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-63da37-d6bc764c6-qtzsp"] Jul 06 23:00:09.004295 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:09.004262 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d989c08-9187-4e74-ba47-0dab96b7e113" path="/var/lib/kubelet/pods/9d989c08-9187-4e74-ba47-0dab96b7e113/volumes" Jul 06 23:00:16.796123 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.796083 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw"] Jul 06 23:00:16.796794 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.796774 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" containerName="maas-api" Jul 06 23:00:16.796842 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.796799 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" containerName="maas-api" Jul 06 23:00:16.796842 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.796837 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d989c08-9187-4e74-ba47-0dab96b7e113" containerName="maas-api" Jul 06 23:00:16.796906 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.796845 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d989c08-9187-4e74-ba47-0dab96b7e113" containerName="maas-api" Jul 06 23:00:16.796975 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.796963 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d989c08-9187-4e74-ba47-0dab96b7e113" containerName="maas-api" Jul 06 23:00:16.797009 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.796984 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99caa7c9-fb85-4f94-be50-b0f8b4b8acc1" containerName="maas-api" Jul 06 23:00:16.800474 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.800454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.802765 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.802747 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-4a6cf6-serving-cert\"" Jul 06 23:00:16.806669 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.806643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-maas-api-tls\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.806933 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.806897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.807046 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.806935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkfk\" (UniqueName: \"kubernetes.io/projected/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-kube-api-access-qhkfk\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.812876 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.812853 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw"] Jul 06 23:00:16.908282 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.908252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-maas-api-tls\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.908472 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.908300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.908472 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.908320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkfk\" (UniqueName: \"kubernetes.io/projected/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-kube-api-access-qhkfk\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.908905 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.908882 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.910853 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.910826 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-maas-api-tls\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:16.919158 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:16.919135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkfk\" (UniqueName: \"kubernetes.io/projected/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-kube-api-access-qhkfk\") pod \"maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:17.114359 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:17.114266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:17.260023 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:17.259997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw"] Jul 06 23:00:17.262751 ip-10-0-136-56 kubenswrapper[2580]: W0706 23:00:17.262719 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334e140f_b6ea_4b52_9e3a_80b40dd69b5b.slice/crio-03630564bd2da1035dc06d796da1cfb3b475e50a4d7bfc9601379438c7448097 WatchSource:0}: Error finding container 03630564bd2da1035dc06d796da1cfb3b475e50a4d7bfc9601379438c7448097: Status 404 returned error can't find the container with id 03630564bd2da1035dc06d796da1cfb3b475e50a4d7bfc9601379438c7448097 Jul 06 23:00:18.112653 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:18.112570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" event={"ID":"334e140f-b6ea-4b52-9e3a-80b40dd69b5b","Type":"ContainerStarted","Data":"c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753"} Jul 06 23:00:18.112653 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:18.112620 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" event={"ID":"334e140f-b6ea-4b52-9e3a-80b40dd69b5b","Type":"ContainerStarted","Data":"03630564bd2da1035dc06d796da1cfb3b475e50a4d7bfc9601379438c7448097"} Jul 06 23:00:18.112653 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:18.112650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:18.131104 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:18.131042 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" podStartSLOduration=1.6486349040000001 podStartE2EDuration="2.131021539s" totalImagesPullingTime="482.386635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:00:18.129692926 +0000 UTC m=+1953.739694132" watchObservedRunningTime="2026-07-06 23:00:18.131021539 +0000 UTC m=+1953.741022756" Jul 06 23:00:24.123343 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:24.123315 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:24.897776 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:24.897744 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5"] Jul 06 23:00:24.901611 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:24.901591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:24.904003 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:24.903976 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-280980-serving-cert\"" Jul 06 23:00:24.911996 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:24.911973 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5"] Jul 06 23:00:24.992400 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:24.992365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33897570-ebc7-40cc-bd95-a21bfdedd24f-maas-api-tls\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:24.992571 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:24.992413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvht\" (UniqueName: \"kubernetes.io/projected/33897570-ebc7-40cc-bd95-a21bfdedd24f-kube-api-access-spvht\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:24.992571 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:24.992464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33897570-ebc7-40cc-bd95-a21bfdedd24f-openshift-service-ca\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:25.093110 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:25.093075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvht\" (UniqueName: \"kubernetes.io/projected/33897570-ebc7-40cc-bd95-a21bfdedd24f-kube-api-access-spvht\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:25.093334 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:25.093124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33897570-ebc7-40cc-bd95-a21bfdedd24f-openshift-service-ca\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:25.093475 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:25.093450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33897570-ebc7-40cc-bd95-a21bfdedd24f-maas-api-tls\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:25.093881 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:25.093861 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33897570-ebc7-40cc-bd95-a21bfdedd24f-openshift-service-ca\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:25.095907 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:25.095886 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33897570-ebc7-40cc-bd95-a21bfdedd24f-maas-api-tls\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:25.100412 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:25.100391 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvht\" (UniqueName: \"kubernetes.io/projected/33897570-ebc7-40cc-bd95-a21bfdedd24f-kube-api-access-spvht\") pod \"maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:25.214545 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:25.214503 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:25.362009 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:25.361986 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5"] Jul 06 23:00:25.364116 ip-10-0-136-56 kubenswrapper[2580]: W0706 23:00:25.364084 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33897570_ebc7_40cc_bd95_a21bfdedd24f.slice/crio-1c7ae2d5c9d12e96fba7820c96a3b37c4eabab8231ab7c1256059486ccf9dcc8 WatchSource:0}: Error finding container 1c7ae2d5c9d12e96fba7820c96a3b37c4eabab8231ab7c1256059486ccf9dcc8: Status 404 returned error can't find the container with id 1c7ae2d5c9d12e96fba7820c96a3b37c4eabab8231ab7c1256059486ccf9dcc8 Jul 06 23:00:26.157957 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:26.157870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" event={"ID":"33897570-ebc7-40cc-bd95-a21bfdedd24f","Type":"ContainerStarted","Data":"67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e"} Jul 06 23:00:26.157957 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:26.157934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" event={"ID":"33897570-ebc7-40cc-bd95-a21bfdedd24f","Type":"ContainerStarted","Data":"1c7ae2d5c9d12e96fba7820c96a3b37c4eabab8231ab7c1256059486ccf9dcc8"} Jul 06 23:00:26.158141 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:26.157982 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:26.176000 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:26.175956 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" podStartSLOduration=1.764370027 podStartE2EDuration="2.175941127s" totalImagesPullingTime="411.5711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:00:26.173735138 +0000 UTC m=+1961.783736347" watchObservedRunningTime="2026-07-06 23:00:26.175941127 +0000 UTC m=+1961.785942369" Jul 06 23:00:32.169091 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:32.169060 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:38.168206 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.168100 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw"] Jul 06 23:00:38.169091 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.169051 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" podUID="334e140f-b6ea-4b52-9e3a-80b40dd69b5b" containerName="maas-api" containerID="cri-o://c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753" gracePeriod=30 Jul 06 23:00:38.417850 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.417826 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:38.535450 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.535421 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-openshift-service-ca\") pod \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " Jul 06 23:00:38.535607 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.535519 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-maas-api-tls\") pod \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " Jul 06 23:00:38.535607 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.535586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhkfk\" (UniqueName: \"kubernetes.io/projected/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-kube-api-access-qhkfk\") pod \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\" (UID: \"334e140f-b6ea-4b52-9e3a-80b40dd69b5b\") " Jul 06 23:00:38.535808 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.535786 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "334e140f-b6ea-4b52-9e3a-80b40dd69b5b" (UID: "334e140f-b6ea-4b52-9e3a-80b40dd69b5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:00:38.535942 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.535916 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:38.537735 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.537692 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-kube-api-access-qhkfk" (OuterVolumeSpecName: "kube-api-access-qhkfk") pod "334e140f-b6ea-4b52-9e3a-80b40dd69b5b" (UID: "334e140f-b6ea-4b52-9e3a-80b40dd69b5b"). InnerVolumeSpecName "kube-api-access-qhkfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:38.537857 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.537757 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "334e140f-b6ea-4b52-9e3a-80b40dd69b5b" (UID: "334e140f-b6ea-4b52-9e3a-80b40dd69b5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:00:38.636609 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.636580 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:38.636609 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:38.636610 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhkfk\" (UniqueName: \"kubernetes.io/projected/334e140f-b6ea-4b52-9e3a-80b40dd69b5b-kube-api-access-qhkfk\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:39.228087 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.228055 2580 generic.go:356] "Generic (PLEG): container finished" podID="334e140f-b6ea-4b52-9e3a-80b40dd69b5b" containerID="c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753" exitCode=0 Jul 06 23:00:39.228531 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.228096 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" Jul 06 23:00:39.228531 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.228135 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" event={"ID":"334e140f-b6ea-4b52-9e3a-80b40dd69b5b","Type":"ContainerDied","Data":"c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753"} Jul 06 23:00:39.228531 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.228209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw" event={"ID":"334e140f-b6ea-4b52-9e3a-80b40dd69b5b","Type":"ContainerDied","Data":"03630564bd2da1035dc06d796da1cfb3b475e50a4d7bfc9601379438c7448097"} Jul 06 23:00:39.228531 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.228234 2580 scope.go:117] "RemoveContainer" containerID="c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753" Jul 06 23:00:39.238481 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.238457 2580 scope.go:117] "RemoveContainer" containerID="c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753" Jul 06 23:00:39.238797 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:00:39.238780 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753\": container with ID starting with c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753 not found: ID does not exist" containerID="c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753" Jul 06 23:00:39.238873 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.238821 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753"} err="failed to get container status \"c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753\": rpc error: code = NotFound desc = could not find container \"c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753\": container with ID starting with c95da2cfa432f33c2a98649098c4d369ce2a0ff0f0823ff8317dbefa6b021753 not found: ID does not exist" Jul 06 23:00:39.245429 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.245391 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw"] Jul 06 23:00:39.247523 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:39.247502 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4a6cf6-5679d4c956-dgfdw"] Jul 06 23:00:41.004262 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:41.004223 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334e140f-b6ea-4b52-9e3a-80b40dd69b5b" path="/var/lib/kubelet/pods/334e140f-b6ea-4b52-9e3a-80b40dd69b5b/volumes" Jul 06 23:00:45.198527 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.198481 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5"] Jul 06 23:00:45.198973 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.198767 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" podUID="33897570-ebc7-40cc-bd95-a21bfdedd24f" containerName="maas-api" containerID="cri-o://67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e" gracePeriod=30 Jul 06 23:00:45.462322 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.462297 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:45.605617 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.605578 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33897570-ebc7-40cc-bd95-a21bfdedd24f-openshift-service-ca\") pod \"33897570-ebc7-40cc-bd95-a21bfdedd24f\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " Jul 06 23:00:45.605837 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.605631 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spvht\" (UniqueName: \"kubernetes.io/projected/33897570-ebc7-40cc-bd95-a21bfdedd24f-kube-api-access-spvht\") pod \"33897570-ebc7-40cc-bd95-a21bfdedd24f\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " Jul 06 23:00:45.605837 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.605685 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33897570-ebc7-40cc-bd95-a21bfdedd24f-maas-api-tls\") pod \"33897570-ebc7-40cc-bd95-a21bfdedd24f\" (UID: \"33897570-ebc7-40cc-bd95-a21bfdedd24f\") " Jul 06 23:00:45.606049 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.606025 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33897570-ebc7-40cc-bd95-a21bfdedd24f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33897570-ebc7-40cc-bd95-a21bfdedd24f" (UID: "33897570-ebc7-40cc-bd95-a21bfdedd24f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:00:45.608097 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.608071 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33897570-ebc7-40cc-bd95-a21bfdedd24f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33897570-ebc7-40cc-bd95-a21bfdedd24f" (UID: "33897570-ebc7-40cc-bd95-a21bfdedd24f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:00:45.608282 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.608264 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33897570-ebc7-40cc-bd95-a21bfdedd24f-kube-api-access-spvht" (OuterVolumeSpecName: "kube-api-access-spvht") pod "33897570-ebc7-40cc-bd95-a21bfdedd24f" (UID: "33897570-ebc7-40cc-bd95-a21bfdedd24f"). InnerVolumeSpecName "kube-api-access-spvht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:45.707239 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.707206 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33897570-ebc7-40cc-bd95-a21bfdedd24f-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:45.707239 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.707233 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spvht\" (UniqueName: \"kubernetes.io/projected/33897570-ebc7-40cc-bd95-a21bfdedd24f-kube-api-access-spvht\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:45.707239 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:45.707244 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33897570-ebc7-40cc-bd95-a21bfdedd24f-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:00:46.269128 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.269095 2580 generic.go:356] "Generic (PLEG): container finished" podID="33897570-ebc7-40cc-bd95-a21bfdedd24f" containerID="67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e" exitCode=0 Jul 06 23:00:46.269631 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.269148 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" Jul 06 23:00:46.269631 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.269190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" event={"ID":"33897570-ebc7-40cc-bd95-a21bfdedd24f","Type":"ContainerDied","Data":"67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e"} Jul 06 23:00:46.269631 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.269250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5" event={"ID":"33897570-ebc7-40cc-bd95-a21bfdedd24f","Type":"ContainerDied","Data":"1c7ae2d5c9d12e96fba7820c96a3b37c4eabab8231ab7c1256059486ccf9dcc8"} Jul 06 23:00:46.269631 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.269273 2580 scope.go:117] "RemoveContainer" containerID="67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e" Jul 06 23:00:46.278352 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.278334 2580 scope.go:117] "RemoveContainer" containerID="67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e" Jul 06 23:00:46.278666 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:00:46.278617 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e\": container with ID starting with 67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e not found: ID does not exist" containerID="67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e" Jul 06 23:00:46.278666 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.278645 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e"} err="failed to get container status \"67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e\": rpc error: code = NotFound desc = could not find container \"67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e\": container with ID starting with 67ae10e04fc0093d99dff680b3fb9b68daabef332f7d67dfdcbf963f3077ee5e not found: ID does not exist" Jul 06 23:00:46.290738 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.290710 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5"] Jul 06 23:00:46.292621 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:46.292601 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-280980-fd6dfb9cd-sjvv5"] Jul 06 23:00:47.005085 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:47.005048 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33897570-ebc7-40cc-bd95-a21bfdedd24f" path="/var/lib/kubelet/pods/33897570-ebc7-40cc-bd95-a21bfdedd24f/volumes" Jul 06 23:00:55.010041 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.010011 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5"] Jul 06 23:00:55.010598 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.010494 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334e140f-b6ea-4b52-9e3a-80b40dd69b5b" containerName="maas-api" Jul 06 23:00:55.010598 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.010507 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="334e140f-b6ea-4b52-9e3a-80b40dd69b5b" containerName="maas-api" Jul 06 23:00:55.010598 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.010525 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33897570-ebc7-40cc-bd95-a21bfdedd24f" containerName="maas-api" Jul 06 23:00:55.010598 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.010533 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33897570-ebc7-40cc-bd95-a21bfdedd24f" containerName="maas-api" Jul 06 23:00:55.010832 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.010633 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33897570-ebc7-40cc-bd95-a21bfdedd24f" containerName="maas-api" Jul 06 23:00:55.010832 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.010642 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="334e140f-b6ea-4b52-9e3a-80b40dd69b5b" containerName="maas-api" Jul 06 23:00:55.013738 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.013709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.016307 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.016271 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-2ea5e8-serving-cert\"" Jul 06 23:00:55.023809 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.023219 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5"] Jul 06 23:00:55.100605 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.100569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.100783 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.100644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b06e4f6-85ba-48aa-aaba-a9701a81386c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.100783 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.100685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2f4\" (UniqueName: \"kubernetes.io/projected/4b06e4f6-85ba-48aa-aaba-a9701a81386c-kube-api-access-7n2f4\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.201385 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.201352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2f4\" (UniqueName: \"kubernetes.io/projected/4b06e4f6-85ba-48aa-aaba-a9701a81386c-kube-api-access-7n2f4\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.201570 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.201438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.201570 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.201484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b06e4f6-85ba-48aa-aaba-a9701a81386c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.201662 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:00:55.201575 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-2ea5e8-serving-cert: secret "maas-api-e2e-shared-a-2ea5e8-serving-cert" not found Jul 06 23:00:55.201662 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:00:55.201659 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls podName:4b06e4f6-85ba-48aa-aaba-a9701a81386c nodeName:}" failed. No retries permitted until 2026-07-06 23:00:55.701637496 +0000 UTC m=+1991.311638693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls") pod "maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" (UID: "4b06e4f6-85ba-48aa-aaba-a9701a81386c") : secret "maas-api-e2e-shared-a-2ea5e8-serving-cert" not found Jul 06 23:00:55.202070 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.202052 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b06e4f6-85ba-48aa-aaba-a9701a81386c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.217882 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.217846 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2f4\" (UniqueName: \"kubernetes.io/projected/4b06e4f6-85ba-48aa-aaba-a9701a81386c-kube-api-access-7n2f4\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.705828 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.705795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.708380 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.708348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls\") pod \"maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:55.928150 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:55.928108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:56.071206 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:56.071158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5"] Jul 06 23:00:56.073515 ip-10-0-136-56 kubenswrapper[2580]: W0706 23:00:56.073491 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b06e4f6_85ba_48aa_aaba_a9701a81386c.slice/crio-ddfc7d959e25df360e728bbd2065c08e980ba8b7737713bcc3d6af4058b1ff0d WatchSource:0}: Error finding container ddfc7d959e25df360e728bbd2065c08e980ba8b7737713bcc3d6af4058b1ff0d: Status 404 returned error can't find the container with id ddfc7d959e25df360e728bbd2065c08e980ba8b7737713bcc3d6af4058b1ff0d Jul 06 23:00:56.323260 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:56.323159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" event={"ID":"4b06e4f6-85ba-48aa-aaba-a9701a81386c","Type":"ContainerStarted","Data":"ddfc7d959e25df360e728bbd2065c08e980ba8b7737713bcc3d6af4058b1ff0d"} Jul 06 23:00:57.329953 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:57.329914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" event={"ID":"4b06e4f6-85ba-48aa-aaba-a9701a81386c","Type":"ContainerStarted","Data":"2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d"} Jul 06 23:00:57.330387 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:57.329981 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:00:57.347697 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:00:57.347644 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" podStartSLOduration=2.722283109 podStartE2EDuration="3.347601195s" totalImagesPullingTime="625.318086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:00:57.346360857 +0000 UTC m=+1992.956362067" watchObservedRunningTime="2026-07-06 23:00:57.347601195 +0000 UTC m=+1992.957602408" Jul 06 23:01:03.078462 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.078424 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq"] Jul 06 23:01:03.087811 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.087779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.090821 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.090793 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-2de350-serving-cert\"" Jul 06 23:01:03.093555 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.093528 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq"] Jul 06 23:01:03.179708 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.179667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xkml\" (UniqueName: \"kubernetes.io/projected/5bbe9669-266a-454a-b48d-a9f93d21bb36-kube-api-access-9xkml\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.179886 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.179718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.179886 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.179818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe9669-266a-454a-b48d-a9f93d21bb36-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.280930 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.280886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.281132 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.281002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe9669-266a-454a-b48d-a9f93d21bb36-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.281132 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:01:03.281050 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-2de350-serving-cert: secret "maas-api-e2e-shared-b-2de350-serving-cert" not found Jul 06 23:01:03.281132 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.281083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xkml\" (UniqueName: \"kubernetes.io/projected/5bbe9669-266a-454a-b48d-a9f93d21bb36-kube-api-access-9xkml\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.281132 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:01:03.281111 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls podName:5bbe9669-266a-454a-b48d-a9f93d21bb36 nodeName:}" failed. No retries permitted until 2026-07-06 23:01:03.781095065 +0000 UTC m=+1999.391096252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls") pod "maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" (UID: "5bbe9669-266a-454a-b48d-a9f93d21bb36") : secret "maas-api-e2e-shared-b-2de350-serving-cert" not found Jul 06 23:01:03.281664 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.281642 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe9669-266a-454a-b48d-a9f93d21bb36-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.290255 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.290226 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xkml\" (UniqueName: \"kubernetes.io/projected/5bbe9669-266a-454a-b48d-a9f93d21bb36-kube-api-access-9xkml\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.341266 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.341188 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:01:03.786903 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.786860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:03.789570 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:03.789531 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls\") pod \"maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:04.002965 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:04.002928 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:04.147464 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:04.147439 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq"] Jul 06 23:01:04.149106 ip-10-0-136-56 kubenswrapper[2580]: W0706 23:01:04.149075 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbe9669_266a_454a_b48d_a9f93d21bb36.slice/crio-33771f0461fef1ae63d1db42514e8aea78ce80db735b6b3f0abd9b80f71e6b7c WatchSource:0}: Error finding container 33771f0461fef1ae63d1db42514e8aea78ce80db735b6b3f0abd9b80f71e6b7c: Status 404 returned error can't find the container with id 33771f0461fef1ae63d1db42514e8aea78ce80db735b6b3f0abd9b80f71e6b7c Jul 06 23:01:04.368664 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:04.368570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" event={"ID":"5bbe9669-266a-454a-b48d-a9f93d21bb36","Type":"ContainerStarted","Data":"33771f0461fef1ae63d1db42514e8aea78ce80db735b6b3f0abd9b80f71e6b7c"} Jul 06 23:01:05.375475 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:05.375433 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" event={"ID":"5bbe9669-266a-454a-b48d-a9f93d21bb36","Type":"ContainerStarted","Data":"580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda"} Jul 06 23:01:05.375950 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:05.375499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:01:05.391919 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:05.391865 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" podStartSLOduration=1.828270473 podStartE2EDuration="2.391845845s" totalImagesPullingTime="563.575372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:01:05.391836978 +0000 UTC m=+2001.001838188" watchObservedRunningTime="2026-07-06 23:01:05.391845845 +0000 UTC m=+2001.001847055" Jul 06 23:01:11.385708 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:01:11.385682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:06:51.242749 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.242712 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5"] Jul 06 23:06:51.243608 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.243574 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" podUID="4b06e4f6-85ba-48aa-aaba-a9701a81386c" containerName="maas-api" containerID="cri-o://2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d" gracePeriod=30 Jul 06 23:06:51.502290 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.502229 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:06:51.591203 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.591155 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b06e4f6-85ba-48aa-aaba-a9701a81386c-openshift-service-ca\") pod \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " Jul 06 23:06:51.591383 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.591299 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n2f4\" (UniqueName: \"kubernetes.io/projected/4b06e4f6-85ba-48aa-aaba-a9701a81386c-kube-api-access-7n2f4\") pod \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " Jul 06 23:06:51.591383 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.591352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls\") pod \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\" (UID: \"4b06e4f6-85ba-48aa-aaba-a9701a81386c\") " Jul 06 23:06:51.591526 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.591500 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b06e4f6-85ba-48aa-aaba-a9701a81386c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b06e4f6-85ba-48aa-aaba-a9701a81386c" (UID: "4b06e4f6-85ba-48aa-aaba-a9701a81386c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:06:51.591697 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.591681 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b06e4f6-85ba-48aa-aaba-a9701a81386c-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:06:51.593257 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.593234 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b06e4f6-85ba-48aa-aaba-a9701a81386c-kube-api-access-7n2f4" (OuterVolumeSpecName: "kube-api-access-7n2f4") pod "4b06e4f6-85ba-48aa-aaba-a9701a81386c" (UID: "4b06e4f6-85ba-48aa-aaba-a9701a81386c"). InnerVolumeSpecName "kube-api-access-7n2f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:06:51.593356 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.593340 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b06e4f6-85ba-48aa-aaba-a9701a81386c" (UID: "4b06e4f6-85ba-48aa-aaba-a9701a81386c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:06:51.693053 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.693019 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n2f4\" (UniqueName: \"kubernetes.io/projected/4b06e4f6-85ba-48aa-aaba-a9701a81386c-kube-api-access-7n2f4\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:06:51.693053 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:51.693046 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b06e4f6-85ba-48aa-aaba-a9701a81386c-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:06:52.214270 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.214235 2580 generic.go:356] "Generic (PLEG): container finished" podID="4b06e4f6-85ba-48aa-aaba-a9701a81386c" containerID="2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d" exitCode=0 Jul 06 23:06:52.214442 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.214281 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" Jul 06 23:06:52.214442 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.214315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" event={"ID":"4b06e4f6-85ba-48aa-aaba-a9701a81386c","Type":"ContainerDied","Data":"2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d"} Jul 06 23:06:52.214442 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.214365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5" event={"ID":"4b06e4f6-85ba-48aa-aaba-a9701a81386c","Type":"ContainerDied","Data":"ddfc7d959e25df360e728bbd2065c08e980ba8b7737713bcc3d6af4058b1ff0d"} Jul 06 23:06:52.214442 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.214387 2580 scope.go:117] "RemoveContainer" containerID="2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d" Jul 06 23:06:52.223722 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.223705 2580 scope.go:117] "RemoveContainer" containerID="2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d" Jul 06 23:06:52.223965 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:06:52.223943 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d\": container with ID starting with 2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d not found: ID does not exist" containerID="2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d" Jul 06 23:06:52.224028 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.223977 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d"} err="failed to get container status \"2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d\": rpc error: code = NotFound desc = could not find container \"2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d\": container with ID starting with 2dc1f4f6e0351f465560bb0fbb942029cabef24544e9a6145cd598520aadd90d not found: ID does not exist" Jul 06 23:06:52.249710 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.249683 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5"] Jul 06 23:06:52.262254 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:52.262226 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ea5e8-f4999f98f-d5gn5"] Jul 06 23:06:53.003571 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:06:53.003539 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b06e4f6-85ba-48aa-aaba-a9701a81386c" path="/var/lib/kubelet/pods/4b06e4f6-85ba-48aa-aaba-a9701a81386c/volumes" Jul 06 23:07:34.493680 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.493646 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq"] Jul 06 23:07:34.494928 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.493911 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" podUID="5bbe9669-266a-454a-b48d-a9f93d21bb36" containerName="maas-api" containerID="cri-o://580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda" gracePeriod=30 Jul 06 23:07:34.751864 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.751801 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:07:34.825032 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.825001 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xkml\" (UniqueName: \"kubernetes.io/projected/5bbe9669-266a-454a-b48d-a9f93d21bb36-kube-api-access-9xkml\") pod \"5bbe9669-266a-454a-b48d-a9f93d21bb36\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " Jul 06 23:07:34.825257 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.825042 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls\") pod \"5bbe9669-266a-454a-b48d-a9f93d21bb36\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " Jul 06 23:07:34.825257 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.825072 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe9669-266a-454a-b48d-a9f93d21bb36-openshift-service-ca\") pod \"5bbe9669-266a-454a-b48d-a9f93d21bb36\" (UID: \"5bbe9669-266a-454a-b48d-a9f93d21bb36\") " Jul 06 23:07:34.825438 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.825415 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbe9669-266a-454a-b48d-a9f93d21bb36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5bbe9669-266a-454a-b48d-a9f93d21bb36" (UID: "5bbe9669-266a-454a-b48d-a9f93d21bb36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:07:34.827198 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.827150 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5bbe9669-266a-454a-b48d-a9f93d21bb36" (UID: "5bbe9669-266a-454a-b48d-a9f93d21bb36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:07:34.827307 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.827239 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbe9669-266a-454a-b48d-a9f93d21bb36-kube-api-access-9xkml" (OuterVolumeSpecName: "kube-api-access-9xkml") pod "5bbe9669-266a-454a-b48d-a9f93d21bb36" (UID: "5bbe9669-266a-454a-b48d-a9f93d21bb36"). InnerVolumeSpecName "kube-api-access-9xkml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:07:34.926141 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.926097 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bbe9669-266a-454a-b48d-a9f93d21bb36-maas-api-tls\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:07:34.926141 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.926128 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe9669-266a-454a-b48d-a9f93d21bb36-openshift-service-ca\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:07:34.926141 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:34.926139 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xkml\" (UniqueName: \"kubernetes.io/projected/5bbe9669-266a-454a-b48d-a9f93d21bb36-kube-api-access-9xkml\") on node \"ip-10-0-136-56.ec2.internal\" DevicePath \"\"" Jul 06 23:07:35.439209 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.439152 2580 generic.go:356] "Generic (PLEG): container finished" podID="5bbe9669-266a-454a-b48d-a9f93d21bb36" containerID="580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda" exitCode=0 Jul 06 23:07:35.439633 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.439608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" event={"ID":"5bbe9669-266a-454a-b48d-a9f93d21bb36","Type":"ContainerDied","Data":"580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda"} Jul 06 23:07:35.439880 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.439863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" event={"ID":"5bbe9669-266a-454a-b48d-a9f93d21bb36","Type":"ContainerDied","Data":"33771f0461fef1ae63d1db42514e8aea78ce80db735b6b3f0abd9b80f71e6b7c"} Jul 06 23:07:35.440029 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.440016 2580 scope.go:117] "RemoveContainer" containerID="580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda" Jul 06 23:07:35.440361 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.440343 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq" Jul 06 23:07:35.450585 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.450559 2580 scope.go:117] "RemoveContainer" containerID="580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda" Jul 06 23:07:35.450871 ip-10-0-136-56 kubenswrapper[2580]: E0706 23:07:35.450853 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda\": container with ID starting with 580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda not found: ID does not exist" containerID="580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda" Jul 06 23:07:35.450940 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.450882 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda"} err="failed to get container status \"580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda\": rpc error: code = NotFound desc = could not find container \"580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda\": container with ID starting with 580294970e57a4e633c5cd668377171f5ec08d6eb40e5ce8ef9f3762167dcbda not found: ID does not exist" Jul 06 23:07:35.461184 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.461147 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq"] Jul 06 23:07:35.465192 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:35.465147 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-2de350-59cc4d5cbc-jlsbq"] Jul 06 23:07:37.004078 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:37.004045 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bbe9669-266a-454a-b48d-a9f93d21bb36" path="/var/lib/kubelet/pods/5bbe9669-266a-454a-b48d-a9f93d21bb36/volumes" Jul 06 23:07:45.750100 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:45.750070 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-k7h5j_f8c50d40-06ed-4ad5-9df5-abf9c313e5dc/manager/0.log" Jul 06 23:07:45.870472 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:45.870441 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p7g27_e1a6e50e-38c4-4413-a5ae-3f12c45367b3/manager/0.log" Jul 06 23:07:45.994047 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:45.994021 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-7f647c485-55h88_4741aeb8-144e-43d5-a8db-b483000bf901/maas-api/0.log" Jul 06 23:07:46.107712 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:46.107641 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722965-wkqs5_70df8d71-ada9-42b0-95e0-e0a8e0b27c08/cleanup/0.log" Jul 06 23:07:46.343511 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:46.343466 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dcbb958d8-k9mds_db5c4bd2-d144-49b1-9fb5-68f01c0de7d3/manager/0.log" Jul 06 23:07:46.947911 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:46.947882 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8c9b4d57b-wcp4z_ce9a2725-f46b-40b6-9516-0374fe569eb6/copy-manifests/0.log" Jul 06 23:07:46.954807 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:46.954774 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8c9b4d57b-wcp4z_ce9a2725-f46b-40b6-9516-0374fe569eb6/manager/0.log" Jul 06 23:07:47.063828 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:47.063796 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-t2hnm_80cdc8e7-ef09-4a48-9179-475df41fb06c/postgres/0.log" Jul 06 23:07:48.953734 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:48.953701 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vxkkf_1c546060-facb-41ff-9f3f-edfff60cb4d6/manager/0.log" Jul 06 23:07:50.140588 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:50.140564 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6lhlq_ea09af4b-fb01-4a7e-b7ab-912c2d218051/payload-pre-processing/0.log" Jul 06 23:07:50.260007 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:50.259977 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-w5wll_610bee22-67d7-488f-8c2e-4540e738fbdb/payload-processing/0.log" Jul 06 23:07:50.724159 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:50.724129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5fwhf_17c6ddc2-09a8-4561-9510-db903c84ff9a/main/0.log" Jul 06 23:07:50.954545 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:50.954516 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-kmtrx_30a69b2a-fd89-4031-9623-3271fef9999f/main/0.log" Jul 06 23:07:51.182559 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:51.182534 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-h29d7_5b2b906b-8a84-4fb8-b88f-420eaeb4a9da/main/0.log" Jul 06 23:07:57.962704 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:57.962672 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4qqdf_f20455a2-a971-4b8f-bfab-879716005fbf/global-pull-secret-syncer/0.log" Jul 06 23:07:58.123905 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:58.123878 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9fw9r_36b3b092-dfcb-4578-87fb-fcfd9937d07d/konnectivity-agent/0.log" Jul 06 23:07:58.224456 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:07:58.224373 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-56.ec2.internal_0ac28a16a2b80991296137c48fab92c8/haproxy/0.log" Jul 06 23:08:03.771802 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:03.771770 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vxkkf_1c546060-facb-41ff-9f3f-edfff60cb4d6/manager/0.log" Jul 06 23:08:05.442044 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.442015 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57995eeb-c068-4e4e-aa55-0e56ab6cbbdb/alertmanager/0.log" Jul 06 23:08:05.479974 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.479950 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57995eeb-c068-4e4e-aa55-0e56ab6cbbdb/config-reloader/0.log" Jul 06 23:08:05.507291 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.507254 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57995eeb-c068-4e4e-aa55-0e56ab6cbbdb/kube-rbac-proxy-web/0.log" Jul 06 23:08:05.534413 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.534353 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57995eeb-c068-4e4e-aa55-0e56ab6cbbdb/kube-rbac-proxy/0.log" Jul 06 23:08:05.558142 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.558119 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57995eeb-c068-4e4e-aa55-0e56ab6cbbdb/kube-rbac-proxy-metric/0.log" Jul 06 23:08:05.586417 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.586388 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57995eeb-c068-4e4e-aa55-0e56ab6cbbdb/prom-label-proxy/0.log" Jul 06 23:08:05.612212 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.612185 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57995eeb-c068-4e4e-aa55-0e56ab6cbbdb/init-config-reloader/0.log" Jul 06 23:08:05.684346 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.684313 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6q2pj_f0155679-45f4-4ca6-addd-f9cf5b85d1bd/kube-state-metrics/0.log" Jul 06 23:08:05.706622 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.706596 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6q2pj_f0155679-45f4-4ca6-addd-f9cf5b85d1bd/kube-rbac-proxy-main/0.log" Jul 06 23:08:05.727963 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.727939 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6q2pj_f0155679-45f4-4ca6-addd-f9cf5b85d1bd/kube-rbac-proxy-self/0.log" Jul 06 23:08:05.757320 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.757300 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b47fdb9d5-zk8mh_3a594377-16e1-4c25-b888-8ff754086a2a/metrics-server/0.log" Jul 06 23:08:05.787836 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.787773 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-nw5zl_22db243c-957b-406a-b7b8-5da9fa8216d8/monitoring-plugin/0.log" Jul 06 23:08:05.919622 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.919593 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94cww_54117083-55de-423b-8b4b-7c9e01649591/node-exporter/0.log" Jul 06 23:08:05.942040 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.942021 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94cww_54117083-55de-423b-8b4b-7c9e01649591/kube-rbac-proxy/0.log" Jul 06 23:08:05.962608 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:05.962588 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94cww_54117083-55de-423b-8b4b-7c9e01649591/init-textfile/0.log" Jul 06 23:08:06.069396 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.069319 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-t5txl_110b2bb5-1ae2-4d9e-8488-0ae89109feae/kube-rbac-proxy-main/0.log" Jul 06 23:08:06.095716 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.095646 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-t5txl_110b2bb5-1ae2-4d9e-8488-0ae89109feae/kube-rbac-proxy-self/0.log" Jul 06 23:08:06.116981 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.116955 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-t5txl_110b2bb5-1ae2-4d9e-8488-0ae89109feae/openshift-state-metrics/0.log" Jul 06 23:08:06.167319 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.167293 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b7f53a6-4aa2-47f4-afe7-8166466e3758/prometheus/0.log" Jul 06 23:08:06.191951 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.191923 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b7f53a6-4aa2-47f4-afe7-8166466e3758/config-reloader/0.log" Jul 06 23:08:06.217101 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.217080 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b7f53a6-4aa2-47f4-afe7-8166466e3758/thanos-sidecar/0.log" Jul 06 23:08:06.238390 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.238370 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b7f53a6-4aa2-47f4-afe7-8166466e3758/kube-rbac-proxy-web/0.log" Jul 06 23:08:06.262352 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.262327 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b7f53a6-4aa2-47f4-afe7-8166466e3758/kube-rbac-proxy/0.log" Jul 06 23:08:06.286356 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.286336 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b7f53a6-4aa2-47f4-afe7-8166466e3758/kube-rbac-proxy-thanos/0.log" Jul 06 23:08:06.310790 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.310773 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b7f53a6-4aa2-47f4-afe7-8166466e3758/init-config-reloader/0.log" Jul 06 23:08:06.343712 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.343656 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-dkcnw_50e32295-7644-416c-abcb-c39454bea80b/prometheus-operator/0.log" Jul 06 23:08:06.366127 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.366105 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-dkcnw_50e32295-7644-416c-abcb-c39454bea80b/kube-rbac-proxy/0.log" Jul 06 23:08:06.424685 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.424659 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75b787f7bb-kqxsj_b40e2735-9974-4e6b-b6ab-5fab1367703b/telemeter-client/0.log" Jul 06 23:08:06.448836 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.448814 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75b787f7bb-kqxsj_b40e2735-9974-4e6b-b6ab-5fab1367703b/reload/0.log" Jul 06 23:08:06.472603 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.472581 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75b787f7bb-kqxsj_b40e2735-9974-4e6b-b6ab-5fab1367703b/kube-rbac-proxy/0.log" Jul 06 23:08:06.507551 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.507502 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7985b97cb5-5whbv_d95494d6-fb6b-428d-b9cd-a9411a4ec95f/thanos-query/0.log" Jul 06 23:08:06.529644 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.529623 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7985b97cb5-5whbv_d95494d6-fb6b-428d-b9cd-a9411a4ec95f/kube-rbac-proxy-web/0.log" Jul 06 23:08:06.555950 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.555932 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7985b97cb5-5whbv_d95494d6-fb6b-428d-b9cd-a9411a4ec95f/kube-rbac-proxy/0.log" Jul 06 23:08:06.578531 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.578514 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7985b97cb5-5whbv_d95494d6-fb6b-428d-b9cd-a9411a4ec95f/prom-label-proxy/0.log" Jul 06 23:08:06.600598 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.600500 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7985b97cb5-5whbv_d95494d6-fb6b-428d-b9cd-a9411a4ec95f/kube-rbac-proxy-rules/0.log" Jul 06 23:08:06.621422 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:06.621380 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7985b97cb5-5whbv_d95494d6-fb6b-428d-b9cd-a9411a4ec95f/kube-rbac-proxy-metrics/0.log" Jul 06 23:08:08.182668 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.182630 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj"] Jul 06 23:08:08.183092 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.183079 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b06e4f6-85ba-48aa-aaba-a9701a81386c" containerName="maas-api" Jul 06 23:08:08.183092 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.183093 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b06e4f6-85ba-48aa-aaba-a9701a81386c" containerName="maas-api" Jul 06 23:08:08.183197 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.183114 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bbe9669-266a-454a-b48d-a9f93d21bb36" containerName="maas-api" Jul 06 23:08:08.183197 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.183122 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bbe9669-266a-454a-b48d-a9f93d21bb36" containerName="maas-api" Jul 06 23:08:08.183272 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.183215 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b06e4f6-85ba-48aa-aaba-a9701a81386c" containerName="maas-api" Jul 06 23:08:08.183272 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.183230 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bbe9669-266a-454a-b48d-a9f93d21bb36" containerName="maas-api" Jul 06 23:08:08.186430 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.186414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.188996 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.188968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tjzlh\"/\"default-dockercfg-dmwst\"" Jul 06 23:08:08.189968 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.189951 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjzlh\"/\"kube-root-ca.crt\"" Jul 06 23:08:08.190077 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.189989 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjzlh\"/\"openshift-service-ca.crt\"" Jul 06 23:08:08.194771 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.194752 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj"] Jul 06 23:08:08.257117 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.257090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dk7\" (UniqueName: \"kubernetes.io/projected/bf439937-8c70-4704-9897-dc110b7f0b2c-kube-api-access-96dk7\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.257302 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.257131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-lib-modules\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.257302 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.257254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-podres\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.257383 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.257300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-sys\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.257383 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.257321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-proc\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358388 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-lib-modules\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358541 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-podres\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358541 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-sys\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358541 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-proc\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358541 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dk7\" (UniqueName: \"kubernetes.io/projected/bf439937-8c70-4704-9897-dc110b7f0b2c-kube-api-access-96dk7\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358714 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358542 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-lib-modules\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358714 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358562 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-sys\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358714 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358564 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-podres\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.358714 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.358592 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf439937-8c70-4704-9897-dc110b7f0b2c-proc\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.366213 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.366195 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dk7\" (UniqueName: \"kubernetes.io/projected/bf439937-8c70-4704-9897-dc110b7f0b2c-kube-api-access-96dk7\") pod \"perf-node-gather-daemonset-lpbbj\" (UID: \"bf439937-8c70-4704-9897-dc110b7f0b2c\") " pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.498110 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.498020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:08.634382 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.634353 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj"] Jul 06 23:08:08.636140 ip-10-0-136-56 kubenswrapper[2580]: W0706 23:08:08.636112 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf439937_8c70_4704_9897_dc110b7f0b2c.slice/crio-017dbebab41ccf7a5079e8bfc94897afa24750f36155416c9d4092e09ebb09b5 WatchSource:0}: Error finding container 017dbebab41ccf7a5079e8bfc94897afa24750f36155416c9d4092e09ebb09b5: Status 404 returned error can't find the container with id 017dbebab41ccf7a5079e8bfc94897afa24750f36155416c9d4092e09ebb09b5 Jul 06 23:08:08.937047 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:08.937022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-vmt4j_d8f92260-54c5-48bf-aebd-5960e0f85343/download-server/0.log" Jul 06 23:08:09.619970 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:09.619928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" event={"ID":"bf439937-8c70-4704-9897-dc110b7f0b2c","Type":"ContainerStarted","Data":"fcd498576f9001fe200fae8bac207e48d12ee5acb42d6ae794b1c17cb2499dc1"} Jul 06 23:08:09.620490 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:09.620017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" event={"ID":"bf439937-8c70-4704-9897-dc110b7f0b2c","Type":"ContainerStarted","Data":"017dbebab41ccf7a5079e8bfc94897afa24750f36155416c9d4092e09ebb09b5"} Jul 06 23:08:09.620490 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:09.620047 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:09.636926 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:09.636878 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" podStartSLOduration=1.6368636749999999 podStartE2EDuration="1.636863675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:08:09.634926648 +0000 UTC m=+2425.244927858" watchObservedRunningTime="2026-07-06 23:08:09.636863675 +0000 UTC m=+2425.246864921" Jul 06 23:08:10.495273 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:10.495249 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8qbqw_4be3f479-cdd8-4f7f-993f-4fa10aac623e/dns/0.log" Jul 06 23:08:10.516948 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:10.516923 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8qbqw_4be3f479-cdd8-4f7f-993f-4fa10aac623e/kube-rbac-proxy/0.log" Jul 06 23:08:10.657779 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:10.657749 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hjcm5_0eca13d8-da9b-48b9-8ce8-f015af526993/dns-node-resolver/0.log" Jul 06 23:08:11.250378 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:11.250354 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t6mx4_e27b56a0-fd8d-4ee6-ac4b-4ae12c13864e/node-ca/0.log" Jul 06 23:08:12.478926 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:12.478895 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6lhlq_ea09af4b-fb01-4a7e-b7ab-912c2d218051/payload-pre-processing/0.log" Jul 06 23:08:12.525931 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:12.525901 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-w5wll_610bee22-67d7-488f-8c2e-4540e738fbdb/payload-processing/0.log" Jul 06 23:08:13.284433 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:13.284402 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vzb48_f618256d-c844-4d04-9f8f-0c341470f10c/serve-healthcheck-canary/0.log" Jul 06 23:08:13.895270 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:13.895240 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w6g5c_4fd1bf24-9a45-494f-971c-f67d4ab7d3c5/kube-rbac-proxy/0.log" Jul 06 23:08:13.917173 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:13.917132 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w6g5c_4fd1bf24-9a45-494f-971c-f67d4ab7d3c5/exporter/0.log" Jul 06 23:08:13.938517 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:13.938468 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w6g5c_4fd1bf24-9a45-494f-971c-f67d4ab7d3c5/extractor/0.log" Jul 06 23:08:15.634661 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:15.634620 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tjzlh/perf-node-gather-daemonset-lpbbj" Jul 06 23:08:16.018142 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:16.018107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-k7h5j_f8c50d40-06ed-4ad5-9df5-abf9c313e5dc/manager/0.log" Jul 06 23:08:16.158838 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:16.158812 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p7g27_e1a6e50e-38c4-4413-a5ae-3f12c45367b3/manager/0.log" Jul 06 23:08:16.206876 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:16.206846 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-7f647c485-55h88_4741aeb8-144e-43d5-a8db-b483000bf901/maas-api/0.log" Jul 06 23:08:16.246688 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:16.246665 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722965-wkqs5_70df8d71-ada9-42b0-95e0-e0a8e0b27c08/cleanup/0.log" Jul 06 23:08:16.380842 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:16.380767 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dcbb958d8-k9mds_db5c4bd2-d144-49b1-9fb5-68f01c0de7d3/manager/0.log" Jul 06 23:08:16.667636 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:16.667559 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8c9b4d57b-wcp4z_ce9a2725-f46b-40b6-9516-0374fe569eb6/manager/0.log" Jul 06 23:08:16.692238 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:16.692218 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8c9b4d57b-wcp4z_ce9a2725-f46b-40b6-9516-0374fe569eb6/copy-manifests/0.log" Jul 06 23:08:16.719120 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:16.719099 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-t2hnm_80cdc8e7-ef09-4a48-9179-475df41fb06c/postgres/0.log" Jul 06 23:08:18.169514 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:18.169484 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c7f89b564-8nd7c_3d51ade3-a1c4-4c70-8069-bc41c464e585/manager/0.log" Jul 06 23:08:18.534198 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:18.534147 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6lhlq_ea09af4b-fb01-4a7e-b7ab-912c2d218051/payload-pre-processing/0.log" Jul 06 23:08:18.574353 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:18.574327 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-w5wll_610bee22-67d7-488f-8c2e-4540e738fbdb/payload-processing/0.log" Jul 06 23:08:20.025886 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:20.025840 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6lhlq_ea09af4b-fb01-4a7e-b7ab-912c2d218051/payload-pre-processing/0.log" Jul 06 23:08:20.067058 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:20.067028 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-w5wll_610bee22-67d7-488f-8c2e-4540e738fbdb/payload-processing/0.log" Jul 06 23:08:24.547066 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:24.546991 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwl2c_37b1c52d-069c-451a-a666-ea8837bf9f7a/kube-multus-additional-cni-plugins/0.log" Jul 06 23:08:24.572211 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:24.572185 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwl2c_37b1c52d-069c-451a-a666-ea8837bf9f7a/egress-router-binary-copy/0.log" Jul 06 23:08:24.592091 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:24.592065 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwl2c_37b1c52d-069c-451a-a666-ea8837bf9f7a/cni-plugins/0.log" Jul 06 23:08:24.614958 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:24.614939 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwl2c_37b1c52d-069c-451a-a666-ea8837bf9f7a/bond-cni-plugin/0.log" Jul 06 23:08:24.635840 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:24.635820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwl2c_37b1c52d-069c-451a-a666-ea8837bf9f7a/routeoverride-cni/0.log" Jul 06 23:08:24.656148 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:24.656119 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwl2c_37b1c52d-069c-451a-a666-ea8837bf9f7a/whereabouts-cni-bincopy/0.log" Jul 06 23:08:24.677083 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:24.677063 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pwl2c_37b1c52d-069c-451a-a666-ea8837bf9f7a/whereabouts-cni/0.log" Jul 06 23:08:24.889101 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:24.889028 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qmvqb_2a9a3de8-111a-4398-925e-5d7faa235047/kube-multus/0.log" Jul 06 23:08:25.028519 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:25.028490 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hcq92_4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433/network-metrics-daemon/0.log" Jul 06 23:08:25.047418 ip-10-0-136-56 kubenswrapper[2580]: I0706 23:08:25.047391 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hcq92_4e8c2cdb-53ed-4c31-a4b5-b6a1a130c433/kube-rbac-proxy/0.log"