Jul 08 18:17:45.275909 ip-10-0-136-178 systemd[1]: Starting Kubernetes Kubelet... Jul 08 18:17:45.625788 ip-10-0-136-178 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:17:45.625788 ip-10-0-136-178 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 18:17:45.625788 ip-10-0-136-178 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:17:45.625788 ip-10-0-136-178 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 18:17:45.625788 ip-10-0-136-178 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:17:45.626655 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.626598 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 18:17:45.630466 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630452 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:17:45.630466 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630466 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630470 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630474 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630477 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630480 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630483 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630485 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630488 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630491 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630493 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630496 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630498 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630501 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630503 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630506 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630508 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630511 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630515 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630517 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630520 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:17:45.630529 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630522 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630524 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630527 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630530 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630533 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630537 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630541 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630544 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630547 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630549 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630551 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630554 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630557 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630559 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630562 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630564 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630567 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630569 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630572 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:17:45.631006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630574 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630577 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630579 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630582 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630585 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630587 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630590 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630592 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630594 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630597 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630599 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630602 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630604 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630607 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630609 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630612 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630615 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630617 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630620 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630622 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:17:45.631435 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630625 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630627 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630630 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630632 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630635 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630637 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630639 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630642 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630644 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630647 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630649 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630652 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630654 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630656 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630672 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630676 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630680 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630683 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630685 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:17:45.631927 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630688 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630691 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630693 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630696 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630698 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630701 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630703 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630706 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630710 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630712 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630716 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630719 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630722 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630725 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630727 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630730 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630732 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630735 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630737 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:17:45.632360 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.630740 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631367 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631373 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631376 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631378 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631381 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631383 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631387 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631389 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631392 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631394 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631397 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631400 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631403 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631405 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631408 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631412 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631415 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631418 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:17:45.632849 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631421 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631424 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631427 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631430 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631433 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631436 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631439 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631442 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631445 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631447 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631450 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631452 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631454 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631457 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631459 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631462 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631464 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631467 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631470 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631472 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:17:45.633291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631474 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631477 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631479 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631482 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631484 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631488 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631490 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631492 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631495 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631497 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631499 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631502 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631504 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631508 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631512 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631515 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631517 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631519 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631522 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631524 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:17:45.633798 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631527 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631529 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631532 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631534 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631536 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631539 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631541 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631544 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631547 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631549 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631552 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631554 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631556 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631559 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631561 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631565 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631569 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631572 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631574 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:17:45.634263 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631577 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631579 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631581 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631584 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631588 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631592 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631595 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631598 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631602 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631605 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631607 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631610 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631612 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631615 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631617 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631619 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631622 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631624 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631627 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631629 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:17:45.634706 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631632 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631635 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631979 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631984 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631987 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631991 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631995 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.631998 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632001 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632003 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632005 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632008 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632010 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632013 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632015 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632019 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632022 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632024 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632027 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632030 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:17:45.635166 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632033 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632036 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632038 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632040 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632043 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632046 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632048 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632051 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632053 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632056 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632058 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632061 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632063 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632065 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632068 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632070 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632073 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632075 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632078 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632080 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:17:45.635628 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632082 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632085 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632087 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632090 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632092 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632095 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632097 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632099 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632102 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632106 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632109 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632112 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632116 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632118 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632121 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632124 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632127 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632129 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632132 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:17:45.636103 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632134 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632137 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632139 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632142 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632144 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632147 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632149 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632151 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632155 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632157 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632159 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632162 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632164 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632167 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632169 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632171 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632174 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632176 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632179 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:17:45.636541 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632181 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632183 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632186 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632188 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632191 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632195 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632198 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632200 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632203 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632205 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632208 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632210 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632212 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632215 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632217 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632219 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632221 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632224 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632226 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632229 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:17:45.636993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632231 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632234 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632236 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632306 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632320 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632329 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632335 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632341 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632346 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632349 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632353 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632356 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632359 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632362 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632365 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632368 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632372 2565 flags.go:64] FLAG: --cgroup-root="" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632375 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632378 2565 flags.go:64] FLAG: --client-ca-file="" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632381 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632384 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632388 2565 flags.go:64] FLAG: --cluster-domain="" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632390 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632393 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 18:17:45.637452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632396 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632399 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632402 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632405 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632408 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632411 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632414 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632417 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632419 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632422 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632425 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632429 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632432 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632435 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632437 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632440 2565 flags.go:64] FLAG: --enable-server="true" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632444 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632448 2565 flags.go:64] FLAG: --event-burst="100" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632451 2565 flags.go:64] FLAG: --event-qps="50" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632454 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632457 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632460 2565 flags.go:64] FLAG: --eviction-hard="" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632471 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632474 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632476 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 18:17:45.638025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632481 2565 flags.go:64] FLAG: --eviction-soft="" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632483 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632486 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632489 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632492 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632495 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632498 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632501 2565 flags.go:64] FLAG: --feature-gates="" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632504 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632507 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632510 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632513 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632515 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632518 2565 flags.go:64] FLAG: --help="false" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632521 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-136-178.ec2.internal" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632524 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632527 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632529 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632532 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632536 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632538 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632541 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632543 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 18:17:45.638616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632548 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632550 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632553 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632556 2565 flags.go:64] FLAG: --kube-reserved="" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632558 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632561 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632564 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632566 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632569 2565 flags.go:64] FLAG: --lock-file="" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632573 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632575 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632578 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632583 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632586 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632589 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632591 2565 flags.go:64] FLAG: --logging-format="text" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632594 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632597 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632600 2565 flags.go:64] FLAG: --manifest-url="" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632602 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632606 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632608 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632612 2565 flags.go:64] FLAG: --max-pods="110" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632615 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632618 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 18:17:45.639395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632620 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632623 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632626 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632629 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632631 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632637 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632640 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632644 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632647 2565 flags.go:64] FLAG: --pod-cidr="" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632650 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632654 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632668 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632672 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632675 2565 flags.go:64] FLAG: --port="10250" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632677 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632680 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09db4354ed4d56a6f" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632685 2565 flags.go:64] FLAG: --qos-reserved="" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632687 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632690 2565 flags.go:64] FLAG: --register-node="true" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632693 2565 flags.go:64] FLAG: --register-with-taints="" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632696 2565 flags.go:64] FLAG: --registry-burst="10" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632699 2565 flags.go:64] FLAG: --registry-qps="5" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632705 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632708 2565 flags.go:64] FLAG: --reserved-memory="" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632711 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 18:17:45.640286 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632714 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632717 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632719 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632722 2565 flags.go:64] FLAG: --runonce="false" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632725 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632727 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632730 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632733 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632736 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632739 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632741 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632744 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632747 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632749 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632752 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632757 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632760 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632762 2565 flags.go:64] FLAG: --system-cgroups="" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632765 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632770 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632773 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632775 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632778 2565 flags.go:64] FLAG: --tls-min-version="" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632782 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632785 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 18:17:45.640899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632788 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632790 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632793 2565 flags.go:64] FLAG: --v="2" Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632796 2565 flags.go:64] FLAG: --version="false" Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632800 2565 flags.go:64] FLAG: --vmodule="" Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632804 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.632807 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632911 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632914 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632917 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632920 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632923 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632926 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632929 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632931 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632934 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632936 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632939 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632941 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632943 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632946 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:17:45.641475 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632948 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632952 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632954 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632957 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632959 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632962 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632964 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632966 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632969 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632972 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632975 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632978 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632980 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632983 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632985 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632987 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632991 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632995 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.632997 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633000 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:17:45.641993 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633002 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633005 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633007 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633009 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633012 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633014 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633017 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633019 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633022 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633025 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633027 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633030 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633032 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633036 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633038 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633041 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633043 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633046 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633048 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633051 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:17:45.642492 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633054 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633058 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633060 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633062 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633065 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633067 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633070 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633072 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633075 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633077 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633080 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633082 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633085 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633087 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633090 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633092 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633095 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633097 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633099 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633102 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:17:45.642971 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633104 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633106 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633110 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633113 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633116 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633120 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633123 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633125 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633128 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633131 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633134 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633136 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633139 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633142 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633145 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633147 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633151 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633154 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633157 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:17:45.643441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633159 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633162 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633164 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633167 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633170 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.633172 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.633737 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.640787 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.640800 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.641482 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.641489 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641548 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641553 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641557 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:17:45.643898 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641561 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641563 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641566 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641569 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641572 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641574 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641577 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641579 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641582 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641585 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641587 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641590 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641592 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641595 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641598 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641601 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641603 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641605 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641608 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641611 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:17:45.644236 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641614 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641616 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641619 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641621 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641624 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641626 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641629 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641632 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641635 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641637 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641640 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641642 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641644 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641647 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641649 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641651 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641654 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641656 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641674 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:17:45.644703 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641676 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641678 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641681 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641683 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641692 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641695 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641697 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641700 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641702 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641705 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641707 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641709 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641712 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641716 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641719 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641722 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641724 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641727 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641729 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641732 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:17:45.645131 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641735 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641737 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641739 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641742 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641744 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641746 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641749 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641751 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641754 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641756 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641759 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641761 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641763 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641766 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641768 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641771 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641773 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641781 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641783 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641786 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:17:45.645580 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641788 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641790 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641793 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641795 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641798 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641800 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641802 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641805 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641807 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641810 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641812 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641816 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641819 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641822 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641825 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641827 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:17:45.646046 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641830 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.641835 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641948 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641953 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641956 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641958 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641960 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641963 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641965 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641968 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641970 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641972 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641974 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641977 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641986 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:17:45.646419 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641989 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641991 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641993 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641996 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.641999 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642002 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642006 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642009 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642012 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642015 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642017 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642020 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642022 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642025 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642027 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642030 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642032 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642034 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642036 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:17:45.646808 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642039 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642041 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642044 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642046 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642048 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642051 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642053 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642056 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642058 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642060 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642063 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642065 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642067 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642075 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642078 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642080 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642082 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642085 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642088 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642090 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:17:45.647247 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642092 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642095 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642097 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642100 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642102 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642104 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642107 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642109 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642112 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642115 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642118 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642120 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642122 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642125 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642129 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642131 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642134 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642137 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642139 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:17:45.647718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642142 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642144 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642146 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642149 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642151 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642153 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642156 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642164 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642167 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642170 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642172 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642174 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642177 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642179 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642182 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642184 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642187 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642191 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642193 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:17:45.648150 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642196 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642198 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642200 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642203 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642205 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642207 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642210 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642212 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:45.642214 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.642219 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.642315 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.645447 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.647419 2565 server.go:1019] "Starting client certificate rotation" Jul 08 18:17:45.648591 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.647515 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 18:17:45.648922 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.647544 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 18:17:45.649408 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.649396 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 18:17:45.652005 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.651989 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 18:17:45.658995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.658981 2565 log.go:25] "Validated CRI v1 runtime API" Jul 08 18:17:45.660909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.660897 2565 log.go:25] "Validated CRI v1 image API" Jul 08 18:17:45.664096 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.664051 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 18:17:45.667772 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.667749 2565 fs.go:135] Filesystem UUIDs: map[5c426f5f-43fc-4991-9ef5-829650f22a32:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 aa8daa87-406d-45a0-8bfa-ec4215482f1b:/dev/nvme0n1p3] Jul 08 18:17:45.667835 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.667770 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 18:17:45.672435 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.672324 2565 manager.go:216] Machine: {Timestamp:2026-07-08 18:17:45.671883228 +0000 UTC m=+0.295470298 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197282 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a866c87b2b1ab1e8635f4433b5a5e SystemUUID:ec2a866c-87b2-b1ab-1e86-35f4433b5a5e BootID:401b3947-e4e2-4e15-b9df-669e68e49570 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4b:75:a8:2c:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4b:75:a8:2c:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:06:a8:d2:93:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 18:17:45.672435 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.672432 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 18:17:45.672548 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.672501 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 18:17:45.675266 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.675245 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 18:17:45.675417 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.675270 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-178.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 18:17:45.675733 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.675718 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 18:17:45.675886 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.675875 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 18:17:45.675914 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.675889 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 18:17:45.675914 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.675901 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 18:17:45.677380 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.677369 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 18:17:45.678654 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.678644 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 18:17:45.681556 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.681545 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 08 18:17:45.681690 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.681681 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 18:17:45.681720 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.681710 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 08 18:17:45.681745 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.681729 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 18:17:45.681745 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.681738 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 18:17:45.681793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.681769 2565 kubelet.go:398] "Adding apiserver pod source" Jul 08 18:17:45.681793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.681780 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 18:17:45.683313 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.683288 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 18:17:45.683313 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.683313 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 18:17:45.686096 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.686078 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 18:17:45.687860 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.687848 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 18:17:45.687913 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.687876 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 18:17:45.688806 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688796 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 18:17:45.688852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688810 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 18:17:45.688852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688816 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 18:17:45.688852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688821 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 18:17:45.688852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688829 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 18:17:45.688852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688835 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 18:17:45.688852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688843 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 18:17:45.688852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688848 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 18:17:45.689035 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688860 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 18:17:45.689035 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688865 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 18:17:45.689035 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688873 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 18:17:45.689035 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688878 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 18:17:45.689035 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688900 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 18:17:45.689035 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.688906 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 18:17:45.691619 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.691600 2565 server.go:1268] "Started kubelet" Jul 08 18:17:45.691724 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.691669 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 18:17:45.691778 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.691707 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 18:17:45.691824 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.691783 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 18:17:45.692241 ip-10-0-136-178 systemd[1]: Started Kubernetes Kubelet. Jul 08 18:17:45.693555 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.693535 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 08 18:17:45.694083 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.694048 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 18:17:45.697053 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.697034 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 18:17:45.700152 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.700135 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 18:17:45.700152 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.700148 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 18:17:45.700276 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.700218 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:45.700918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.700844 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 18:17:45.700918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.700860 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 18:17:45.701160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701031 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 18:17:45.701160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701075 2565 factory.go:55] Registering systemd factory Jul 08 18:17:45.701160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701089 2565 factory.go:223] Registration of the systemd container factory successfully Jul 08 18:17:45.701160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701111 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 08 18:17:45.701160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701121 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 08 18:17:45.701424 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701330 2565 factory.go:153] Registering CRI-O factory Jul 08 18:17:45.701424 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701345 2565 factory.go:223] Registration of the crio container factory successfully Jul 08 18:17:45.701424 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701395 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 18:17:45.701424 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701416 2565 factory.go:103] Registering Raw factory Jul 08 18:17:45.701723 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.701430 2565 manager.go:1196] Started watching for new ooms in manager Jul 08 18:17:45.702018 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.702002 2565 manager.go:349] Starting recovery of all containers Jul 08 18:17:45.705315 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.705293 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 18:17:45.705556 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.705510 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 18:17:45.706396 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.706366 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 18:17:45.707150 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.706324 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-178.ec2.internal.18c0635431243885 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-178.ec2.internal,UID:ip-10-0-136-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-178.ec2.internal,},FirstTimestamp:2026-07-08 18:17:45.691576453 +0000 UTC m=+0.315163523,LastTimestamp:2026-07-08 18:17:45.691576453 +0000 UTC m=+0.315163523,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-178.ec2.internal,}" Jul 08 18:17:45.707389 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.707335 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 18:17:45.707848 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.707810 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 18:17:45.714762 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.714645 2565 manager.go:354] Recovery completed Jul 08 18:17:45.718527 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.718516 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:17:45.720240 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.720222 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:17:45.720295 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.720255 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:17:45.720295 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.720266 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:17:45.720690 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.720678 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 18:17:45.720690 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.720688 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 18:17:45.720806 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.720698 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:17:45.720806 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.720702 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 08 18:17:45.721321 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.721306 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:17:45.721373 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.721331 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:17:45.721373 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.721339 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:17:45.722009 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.721998 2565 policy_none.go:49] "None policy: Start" Jul 08 18:17:45.722054 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.722013 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 18:17:45.722054 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.722022 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 18:17:45.722511 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.722454 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-178.ec2.internal.18c0635432d98ee0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-178.ec2.internal,UID:ip-10-0-136-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-178.ec2.internal,},FirstTimestamp:2026-07-08 18:17:45.720237792 +0000 UTC m=+0.343824862,LastTimestamp:2026-07-08 18:17:45.720237792 +0000 UTC m=+0.343824862,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-178.ec2.internal,}" Jul 08 18:17:45.723175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.723165 2565 policy_none.go:47] "Start" Jul 08 18:17:45.733961 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.733906 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-178.ec2.internal.18c0635432d9e943 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-178.ec2.internal,UID:ip-10-0-136-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-178.ec2.internal,},FirstTimestamp:2026-07-08 18:17:45.720260931 +0000 UTC m=+0.343848001,LastTimestamp:2026-07-08 18:17:45.720260931 +0000 UTC m=+0.343848001,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-178.ec2.internal,}" Jul 08 18:17:45.738012 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.737997 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qmhj7" Jul 08 18:17:45.745414 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.745376 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qmhj7" Jul 08 18:17:45.747641 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.747585 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-178.ec2.internal.18c0635432da0a95 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-178.ec2.internal,UID:ip-10-0-136-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-178.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-178.ec2.internal,},FirstTimestamp:2026-07-08 18:17:45.720269461 +0000 UTC m=+0.343856531,LastTimestamp:2026-07-08 18:17:45.720269461 +0000 UTC m=+0.343856531,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-178.ec2.internal,}" Jul 08 18:17:45.756210 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.756198 2565 manager.go:337] "Starting Device Plugin manager" Jul 08 18:17:45.756278 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.756227 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 18:17:45.756278 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.756240 2565 server.go:87] "Starting device plugin registration server" Jul 08 18:17:45.756448 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.756439 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 18:17:45.756490 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.756450 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 18:17:45.756566 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.756552 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 18:17:45.756757 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.756715 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 18:17:45.756757 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.756731 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 18:17:45.757266 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.757246 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 18:17:45.757321 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.757296 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:45.857478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.857459 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:17:45.858170 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.858157 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:17:45.858231 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.858184 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:17:45.858231 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.858203 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:17:45.858231 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.858222 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-178.ec2.internal" Jul 08 18:17:45.866737 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.866721 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-178.ec2.internal" Jul 08 18:17:45.877144 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.866747 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-178.ec2.internal\": node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:45.900918 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.900898 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:45.919819 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.919793 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 18:17:45.921009 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.920995 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 18:17:45.921092 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.921035 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 18:17:45.921092 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.921061 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 18:17:45.921200 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:45.921101 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 18:17:45.924279 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:45.924253 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:17:46.004792 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.004772 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:46.022236 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.022213 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal"] Jul 08 18:17:46.022295 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.022274 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:17:46.023757 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.023728 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:17:46.023757 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.023754 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:17:46.023901 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.023763 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:17:46.025276 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.025263 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:17:46.025411 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.025395 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.025449 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.025427 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:17:46.026088 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.026069 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:17:46.026185 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.026078 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:17:46.026185 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.026131 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:17:46.026185 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.026142 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:17:46.026185 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.026105 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:17:46.026185 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.026182 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:17:46.027588 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.027565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.027694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.027594 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:17:46.028539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.028523 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:17:46.028629 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.028550 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:17:46.028629 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.028562 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:17:46.050384 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.050365 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-178.ec2.internal\" not found" node="ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.054930 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.054914 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-178.ec2.internal\" not found" node="ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.105038 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.105013 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:46.105094 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.105059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd528dcab299f7ca40ff67e8a092082-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal\" (UID: \"3dd528dcab299f7ca40ff67e8a092082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.105094 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.105085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd528dcab299f7ca40ff67e8a092082-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal\" (UID: \"3dd528dcab299f7ca40ff67e8a092082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.105166 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.105103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4bc3b05febacfc431990abc062b9833-config\") pod \"kube-apiserver-proxy-ip-10-0-136-178.ec2.internal\" (UID: \"a4bc3b05febacfc431990abc062b9833\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.205802 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.205762 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:46.205851 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.205814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd528dcab299f7ca40ff67e8a092082-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal\" (UID: \"3dd528dcab299f7ca40ff67e8a092082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.205851 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.205839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4bc3b05febacfc431990abc062b9833-config\") pod \"kube-apiserver-proxy-ip-10-0-136-178.ec2.internal\" (UID: \"a4bc3b05febacfc431990abc062b9833\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.205914 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.205854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd528dcab299f7ca40ff67e8a092082-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal\" (UID: \"3dd528dcab299f7ca40ff67e8a092082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.205914 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.205886 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd528dcab299f7ca40ff67e8a092082-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal\" (UID: \"3dd528dcab299f7ca40ff67e8a092082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.205979 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.205917 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd528dcab299f7ca40ff67e8a092082-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal\" (UID: \"3dd528dcab299f7ca40ff67e8a092082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.205979 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.205922 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4bc3b05febacfc431990abc062b9833-config\") pod \"kube-apiserver-proxy-ip-10-0-136-178.ec2.internal\" (UID: \"a4bc3b05febacfc431990abc062b9833\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.307806 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.307783 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:46.354379 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.354353 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.358602 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.358482 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.408040 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.408012 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:46.508979 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.508911 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:46.609123 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.609092 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:46.647517 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.647489 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 18:17:46.647984 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.647610 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 18:17:46.709253 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:46.709230 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-178.ec2.internal\" not found" Jul 08 18:17:46.726646 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.726623 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:17:46.748630 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.748484 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 18:12:46 +0000 UTC" deadline="2028-03-31 07:11:03.665664994 +0000 UTC" Jul 08 18:17:46.748630 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.748622 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15156h53m16.91704991s" Jul 08 18:17:46.776270 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.776210 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:17:46.801346 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.801319 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.815493 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.815474 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 18:17:46.816642 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.816622 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" Jul 08 18:17:46.824555 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:46.824536 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 18:17:47.000958 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:47.000931 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bc3b05febacfc431990abc062b9833.slice/crio-49a479f1227bd9fd9c4f38029d4f1d96a1e20be4273ea3961dc209ddd83a3309 WatchSource:0}: Error finding container 49a479f1227bd9fd9c4f38029d4f1d96a1e20be4273ea3961dc209ddd83a3309: Status 404 returned error can't find the container with id 49a479f1227bd9fd9c4f38029d4f1d96a1e20be4273ea3961dc209ddd83a3309 Jul 08 18:17:47.002992 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:47.002963 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd528dcab299f7ca40ff67e8a092082.slice/crio-1d9c1931cc510c54c752a4d37ff6dfe847e311e8e00f41636720f269370bc9d0 WatchSource:0}: Error finding container 1d9c1931cc510c54c752a4d37ff6dfe847e311e8e00f41636720f269370bc9d0: Status 404 returned error can't find the container with id 1d9c1931cc510c54c752a4d37ff6dfe847e311e8e00f41636720f269370bc9d0 Jul 08 18:17:47.006761 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.006745 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:17:47.101162 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.101112 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:17:47.683441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.683219 2565 apiserver.go:52] "Watching apiserver" Jul 08 18:17:47.693257 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.692973 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 18:17:47.693402 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.693385 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-c228s","openshift-image-registry/node-ca-bjk7q","openshift-multus/multus-8lbkn","openshift-multus/multus-additional-cni-plugins-dzfqj","openshift-multus/network-metrics-daemon-4dl6h","openshift-network-diagnostics/network-check-target-pfzrd","kube-system/global-pull-secret-syncer-v7kp9","kube-system/konnectivity-agent-vldb6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal","openshift-network-operator/iptables-alerter-vflrw","openshift-ovn-kubernetes/ovnkube-node-ggfnv","kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal","openshift-cluster-node-tuning-operator/tuned-42rdp"] Jul 08 18:17:47.694868 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.694847 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.694967 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.694927 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:17:47.696138 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.696117 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.697360 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.697340 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.699188 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.698789 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.699719 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.699377 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:17:47.699719 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.699542 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f5bks\"" Jul 08 18:17:47.699719 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.699543 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 18:17:47.699938 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.699829 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 18:17:47.700347 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.700090 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:47.700347 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.700164 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 18:17:47.700347 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.700171 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 18:17:47.700347 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.700160 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:17:47.700347 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.700245 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 18:17:47.700738 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.700704 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 18:17:47.700825 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.700748 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5jpc2\"" Jul 08 18:17:47.701732 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.701694 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 18:17:47.701732 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.701709 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 18:17:47.701876 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.701814 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r6xvx\"" Jul 08 18:17:47.702226 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.702207 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 18:17:47.702293 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.702249 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:47.702350 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.702323 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:17:47.705085 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.705062 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.706825 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.706805 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.707574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.707554 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 18:17:47.707963 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.707939 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tszg6\"" Jul 08 18:17:47.708062 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.708050 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 18:17:47.708140 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.707950 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 18:17:47.708305 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.708285 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.709696 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.709678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:17:47.710828 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.710809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.712288 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.712271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.713372 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.713353 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 18:17:47.713656 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.713638 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mwshw\"" Jul 08 18:17:47.713721 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.713693 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:17:47.713850 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.713829 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 18:17:47.714071 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714009 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cwg49\"" Jul 08 18:17:47.714154 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714096 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 18:17:47.714154 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714109 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 18:17:47.714256 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714226 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 18:17:47.714349 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714332 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 18:17:47.714403 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714391 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 18:17:47.714456 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714399 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 18:17:47.714538 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714519 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tk92d\"" Jul 08 18:17:47.714616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714604 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mpbz6\"" Jul 08 18:17:47.714708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714684 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 18:17:47.714790 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714774 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 18:17:47.714878 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714858 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7z28\" (UniqueName: \"kubernetes.io/projected/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-kube-api-access-h7z28\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.714934 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-os-release\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.714934 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-conf-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715036 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:47.715036 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.714975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-host-slash\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.715036 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-etc-kubernetes\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715178 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715056 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.715178 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttv2f\" (UniqueName: \"kubernetes.io/projected/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-kube-api-access-ttv2f\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.715178 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715135 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-cni-binary-copy\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715178 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-system-cni-dir\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.715376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cnibin\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.715376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d38d4dce-6648-426f-98f7-d2fbaf4e89df-kubelet-config\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.715376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-system-cni-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-socket-dir-parent\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-multus-certs\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.715665 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715394 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 18:17:47.715665 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715395 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdtkl\" (UniqueName: \"kubernetes.io/projected/79fe673d-c3eb-4eda-8558-6498fbd55bd7-kube-api-access-cdtkl\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:47.715665 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rmf\" (UniqueName: \"kubernetes.io/projected/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-kube-api-access-p2rmf\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715665 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715474 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 18:17:47.715665 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715554 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 18:17:47.715665 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-cnibin\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715665 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d38d4dce-6648-426f-98f7-d2fbaf4e89df-dbus\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-cni-multus\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-daemon-config\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715780 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715790 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7zdck\"" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715804 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-kubelet\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715845 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-os-release\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-cni-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.715991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.715973 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-k8s-cni-cncf-io\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.716377 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.716001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-netns\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.716377 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.716032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-cni-bin\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.716377 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.716072 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-hostroot\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.716504 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.716379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.716504 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.716438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-iptables-alerter-script\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.720280 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.720255 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 18:17:47.743756 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.743716 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-br65g" Jul 08 18:17:47.753357 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.753333 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-br65g" Jul 08 18:17:47.802125 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.802100 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 18:17:47.805551 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.805529 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:17:47.817271 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-multus-certs\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.817370 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.817370 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-socket-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.817370 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-registration-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.817537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-kubelet\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.817537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-tuned\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.817537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rmf\" (UniqueName: \"kubernetes.io/projected/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-kube-api-access-p2rmf\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.817537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817450 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-modprobe-d\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.817537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysctl-conf\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.817537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-etc-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.817537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8cf354c-96c9-4feb-a464-d48b6a8d7fdf-agent-certs\") pod \"konnectivity-agent-vldb6\" (UID: \"f8cf354c-96c9-4feb-a464-d48b6a8d7fdf\") " pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d38d4dce-6648-426f-98f7-d2fbaf4e89df-dbus\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-daemon-config\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-tmp-dir\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-kubelet\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-os-release\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e769af3-83dc-4841-95a4-b514d5e6b179-serviceca\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-netns\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-hostroot\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817873 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwhd\" (UniqueName: \"kubernetes.io/projected/9e769af3-83dc-4841-95a4-b514d5e6b179-kube-api-access-7pwhd\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.817898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-iptables-alerter-script\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-conf-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817936 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-multus-certs\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.817979 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-sys\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-run-netns\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818024 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-ovn\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818061 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-os-release\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818079 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-conf-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818103 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-netns\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818163 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-hostroot\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.818441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818272 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d38d4dce-6648-426f-98f7-d2fbaf4e89df-dbus\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818574 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818600 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818631 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-iptables-alerter-script\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818697 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-kubelet\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttv2f\" (UniqueName: \"kubernetes.io/projected/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-kube-api-access-ttv2f\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.818728 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-env-overrides\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818817 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-daemon-config\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818865 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-system-cni-dir\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.818909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-run\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818947 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-system-cni-dir\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.818952 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret podName:d38d4dce-6648-426f-98f7-d2fbaf4e89df nodeName:}" failed. No retries permitted until 2026-07-08 18:17:48.318902498 +0000 UTC m=+2.942489587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret") pod "global-pull-secret-syncer-v7kp9" (UID: "d38d4dce-6648-426f-98f7-d2fbaf4e89df") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.818991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-system-cni-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-socket-dir-parent\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819065 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-system-cni-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtkl\" (UniqueName: \"kubernetes.io/projected/79fe673d-c3eb-4eda-8558-6498fbd55bd7-kube-api-access-cdtkl\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-lib-modules\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78707327-a0b1-4913-a4ba-6bf98aa3e78c-tmp\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819182 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-slash\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-hosts-file\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819291 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwdcb\" (UniqueName: \"kubernetes.io/projected/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-kube-api-access-cwdcb\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819311 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-socket-dir-parent\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-cnibin\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819396 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-cnibin\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.819444 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysctl-d\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c945l\" (UniqueName: \"kubernetes.io/projected/78707327-a0b1-4913-a4ba-6bf98aa3e78c-kube-api-access-c945l\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-var-lib-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-log-socket\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e769af3-83dc-4841-95a4-b514d5e6b179-host\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-cni-multus\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-sys-fs\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-host\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-systemd\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819659 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-cni-multus\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-systemd-units\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovnkube-script-lib\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-cni-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-k8s-cni-cncf-io\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819844 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.820237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-cni-bin\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-kubernetes\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovnkube-config\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.819916 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7z28\" (UniqueName: \"kubernetes.io/projected/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-kube-api-access-h7z28\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-run-k8s-cni-cncf-io\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.819974 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:17:48.319960883 +0000 UTC m=+2.943547946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819980 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-multus-cni-dir\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.819991 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-host-var-lib-cni-bin\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-os-release\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820184 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-host-slash\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-etc-kubernetes\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820256 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-host-slash\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-device-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820420 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-os-release\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkdnj\" (UniqueName: \"kubernetes.io/projected/bfbd0e88-1340-470e-b342-c106adec2bb5-kube-api-access-kkdnj\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysconfig\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820512 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-etc-kubernetes\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-node-log\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-cni-bin\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-cni-binary-copy\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-systemd\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-var-lib-kubelet\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-cni-netd\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820734 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820759 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovn-node-metrics-cert\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p26s\" (UniqueName: \"kubernetes.io/projected/5926e59d-0612-4f42-bc6c-d45f60b419c3-kube-api-access-4p26s\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8cf354c-96c9-4feb-a464-d48b6a8d7fdf-konnectivity-ca\") pod \"konnectivity-agent-vldb6\" (UID: \"f8cf354c-96c9-4feb-a464-d48b6a8d7fdf\") " pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820842 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cnibin\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d38d4dce-6648-426f-98f7-d2fbaf4e89df-kubelet-config\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820973 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d38d4dce-6648-426f-98f7-d2fbaf4e89df-kubelet-config\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:47.821981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.820978 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-cnibin\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.822898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.821233 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-cni-binary-copy\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.826016 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.825990 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 18:17:47.827394 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.827372 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:17:47.827488 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.827398 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:17:47.827488 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.827411 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n9nhs for pod openshift-network-diagnostics/network-check-target-pfzrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:47.827488 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:47.827477 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs podName:a61af694-9ea9-4f7b-8509-cb463bb5d73b nodeName:}" failed. No retries permitted until 2026-07-08 18:17:48.327460452 +0000 UTC m=+2.951047535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nhs" (UniqueName: "kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs") pod "network-check-target-pfzrd" (UID: "a61af694-9ea9-4f7b-8509-cb463bb5d73b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:47.830525 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.830474 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtkl\" (UniqueName: \"kubernetes.io/projected/79fe673d-c3eb-4eda-8558-6498fbd55bd7-kube-api-access-cdtkl\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:47.831116 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.831075 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttv2f\" (UniqueName: \"kubernetes.io/projected/e7d23d0c-cd9d-43bf-b2bb-569e389eae4b-kube-api-access-ttv2f\") pod \"multus-additional-cni-plugins-dzfqj\" (UID: \"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b\") " pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:47.831116 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.831090 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rmf\" (UniqueName: \"kubernetes.io/projected/8e202106-9ec8-4675-8bd9-f3c3ba45cb3e-kube-api-access-p2rmf\") pod \"multus-8lbkn\" (UID: \"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e\") " pod="openshift-multus/multus-8lbkn" Jul 08 18:17:47.833327 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.833303 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7z28\" (UniqueName: \"kubernetes.io/projected/c39b1b9d-ddb3-4f4e-96c9-67bba8530d59-kube-api-access-h7z28\") pod \"iptables-alerter-vflrw\" (UID: \"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59\") " pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:47.921780 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.921780 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwhd\" (UniqueName: \"kubernetes.io/projected/9e769af3-83dc-4841-95a4-b514d5e6b179-kube-api-access-7pwhd\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.921780 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.921780 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-sys\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-run-netns\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-ovn\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-env-overrides\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-run\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-lib-modules\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78707327-a0b1-4913-a4ba-6bf98aa3e78c-tmp\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-slash\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.921975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-hosts-file\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwdcb\" (UniqueName: \"kubernetes.io/projected/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-kube-api-access-cwdcb\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysctl-d\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922050 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c945l\" (UniqueName: \"kubernetes.io/projected/78707327-a0b1-4913-a4ba-6bf98aa3e78c-kube-api-access-c945l\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-var-lib-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-log-socket\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e769af3-83dc-4841-95a4-b514d5e6b179-host\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-sys-fs\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-host\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.922237 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-systemd\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-systemd-units\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovnkube-script-lib\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-kubernetes\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovnkube-config\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922350 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-slash\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922367 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-run-netns\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-device-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922429 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-ovn\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkdnj\" (UniqueName: \"kubernetes.io/projected/bfbd0e88-1340-470e-b342-c106adec2bb5-kube-api-access-kkdnj\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922476 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysconfig\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-node-log\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-cni-bin\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-systemd\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-var-lib-kubelet\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922690 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-var-lib-kubelet\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922730 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922789 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-sys\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922850 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-kubernetes\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922914 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-log-socket\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922930 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-env-overrides\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922953 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-lib-modules\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.922975 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-hosts-file\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923042 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-run\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923098 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysconfig\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923150 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923200 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-device-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923219 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysctl-d\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923249 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovnkube-script-lib\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923273 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923291 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovnkube-config\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923318 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-cni-bin\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.923478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923369 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-systemd\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923438 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e769af3-83dc-4841-95a4-b514d5e6b179-host\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923440 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-var-lib-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923458 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-node-log\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923480 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-run-systemd\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923502 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-systemd-units\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-cni-netd\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923535 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-sys-fs\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923539 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-cni-netd\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovn-node-metrics-cert\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923593 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923586 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-host\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p26s\" (UniqueName: \"kubernetes.io/projected/5926e59d-0612-4f42-bc6c-d45f60b419c3-kube-api-access-4p26s\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8cf354c-96c9-4feb-a464-d48b6a8d7fdf-konnectivity-ca\") pod \"konnectivity-agent-vldb6\" (UID: \"f8cf354c-96c9-4feb-a464-d48b6a8d7fdf\") " pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-socket-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-registration-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.924362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-kubelet\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-tuned\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-modprobe-d\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923853 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-registration-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysctl-conf\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-etc-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8cf354c-96c9-4feb-a464-d48b6a8d7fdf-agent-certs\") pod \"konnectivity-agent-vldb6\" (UID: \"f8cf354c-96c9-4feb-a464-d48b6a8d7fdf\") " pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-tmp-dir\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.923973 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbd0e88-1340-470e-b342-c106adec2bb5-socket-dir\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.924019 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e769af3-83dc-4841-95a4-b514d5e6b179-serviceca\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.924269 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-tmp-dir\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.924258 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8cf354c-96c9-4feb-a464-d48b6a8d7fdf-konnectivity-ca\") pod \"konnectivity-agent-vldb6\" (UID: \"f8cf354c-96c9-4feb-a464-d48b6a8d7fdf\") " pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.924301 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-modprobe-d\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.924346 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-host-kubelet\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.924439 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-sysctl-conf\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.924478 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e769af3-83dc-4841-95a4-b514d5e6b179-serviceca\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:47.925215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.924494 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5926e59d-0612-4f42-bc6c-d45f60b419c3-etc-openvswitch\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.927256 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.927227 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78707327-a0b1-4913-a4ba-6bf98aa3e78c-etc-tuned\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.927362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.927211 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" event={"ID":"a4bc3b05febacfc431990abc062b9833","Type":"ContainerStarted","Data":"49a479f1227bd9fd9c4f38029d4f1d96a1e20be4273ea3961dc209ddd83a3309"} Jul 08 18:17:47.927892 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.927870 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5926e59d-0612-4f42-bc6c-d45f60b419c3-ovn-node-metrics-cert\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.927892 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.927870 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8cf354c-96c9-4feb-a464-d48b6a8d7fdf-agent-certs\") pod \"konnectivity-agent-vldb6\" (UID: \"f8cf354c-96c9-4feb-a464-d48b6a8d7fdf\") " pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:17:47.929033 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.928998 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78707327-a0b1-4913-a4ba-6bf98aa3e78c-tmp\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.929497 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.929472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" event={"ID":"3dd528dcab299f7ca40ff67e8a092082","Type":"ContainerStarted","Data":"1d9c1931cc510c54c752a4d37ff6dfe847e311e8e00f41636720f269370bc9d0"} Jul 08 18:17:47.939594 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.939341 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p26s\" (UniqueName: \"kubernetes.io/projected/5926e59d-0612-4f42-bc6c-d45f60b419c3-kube-api-access-4p26s\") pod \"ovnkube-node-ggfnv\" (UID: \"5926e59d-0612-4f42-bc6c-d45f60b419c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:47.939594 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.939510 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c945l\" (UniqueName: \"kubernetes.io/projected/78707327-a0b1-4913-a4ba-6bf98aa3e78c-kube-api-access-c945l\") pod \"tuned-42rdp\" (UID: \"78707327-a0b1-4913-a4ba-6bf98aa3e78c\") " pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:47.940195 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.939978 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkdnj\" (UniqueName: \"kubernetes.io/projected/bfbd0e88-1340-470e-b342-c106adec2bb5-kube-api-access-kkdnj\") pod \"aws-ebs-csi-driver-node-gmcwm\" (UID: \"bfbd0e88-1340-470e-b342-c106adec2bb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:47.940195 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.940143 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwdcb\" (UniqueName: \"kubernetes.io/projected/d28adaf6-6f30-4afd-80cf-191bc1b23cc0-kube-api-access-cwdcb\") pod \"node-resolver-c228s\" (UID: \"d28adaf6-6f30-4afd-80cf-191bc1b23cc0\") " pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:47.940454 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:47.940438 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwhd\" (UniqueName: \"kubernetes.io/projected/9e769af3-83dc-4841-95a4-b514d5e6b179-kube-api-access-7pwhd\") pod \"node-ca-bjk7q\" (UID: \"9e769af3-83dc-4841-95a4-b514d5e6b179\") " pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:48.011952 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.011906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vflrw" Jul 08 18:17:48.021787 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.021589 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8lbkn" Jul 08 18:17:48.022006 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.021980 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39b1b9d_ddb3_4f4e_96c9_67bba8530d59.slice/crio-316d9a23a62b1565d8cbf49a6b3aca3061185b473951e37ca7bf711ae7299b42 WatchSource:0}: Error finding container 316d9a23a62b1565d8cbf49a6b3aca3061185b473951e37ca7bf711ae7299b42: Status 404 returned error can't find the container with id 316d9a23a62b1565d8cbf49a6b3aca3061185b473951e37ca7bf711ae7299b42 Jul 08 18:17:48.032507 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.032483 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e202106_9ec8_4675_8bd9_f3c3ba45cb3e.slice/crio-1046453f5bf3bb082c9e81cbfdad44435fcad4cea6fe567dfaa3d876bcbc24de WatchSource:0}: Error finding container 1046453f5bf3bb082c9e81cbfdad44435fcad4cea6fe567dfaa3d876bcbc24de: Status 404 returned error can't find the container with id 1046453f5bf3bb082c9e81cbfdad44435fcad4cea6fe567dfaa3d876bcbc24de Jul 08 18:17:48.033540 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.033472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" Jul 08 18:17:48.040858 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.040807 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" Jul 08 18:17:48.046479 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.046453 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d23d0c_cd9d_43bf_b2bb_569e389eae4b.slice/crio-8a4a10fd10d8b19f2e68c7e40aa1151bb9911c63b9c87caf3f59e6e2b6727551 WatchSource:0}: Error finding container 8a4a10fd10d8b19f2e68c7e40aa1151bb9911c63b9c87caf3f59e6e2b6727551: Status 404 returned error can't find the container with id 8a4a10fd10d8b19f2e68c7e40aa1151bb9911c63b9c87caf3f59e6e2b6727551 Jul 08 18:17:48.051291 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.051212 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbd0e88_1340_470e_b342_c106adec2bb5.slice/crio-7598a25c3ec4526ea25a21dff0a328a0e202585b2ea4070d9253612690346cc7 WatchSource:0}: Error finding container 7598a25c3ec4526ea25a21dff0a328a0e202585b2ea4070d9253612690346cc7: Status 404 returned error can't find the container with id 7598a25c3ec4526ea25a21dff0a328a0e202585b2ea4070d9253612690346cc7 Jul 08 18:17:48.051396 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.051318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-42rdp" Jul 08 18:17:48.059509 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.059485 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjk7q" Jul 08 18:17:48.060077 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.060049 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78707327_a0b1_4913_a4ba_6bf98aa3e78c.slice/crio-75f72121d368aab9f8b6af318d2fb782fcde36c055f4d7ca0e63a98df36997fc WatchSource:0}: Error finding container 75f72121d368aab9f8b6af318d2fb782fcde36c055f4d7ca0e63a98df36997fc: Status 404 returned error can't find the container with id 75f72121d368aab9f8b6af318d2fb782fcde36c055f4d7ca0e63a98df36997fc Jul 08 18:17:48.066769 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.066750 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:17:48.068506 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.068482 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e769af3_83dc_4841_95a4_b514d5e6b179.slice/crio-70e53782a2377ebe858636dfabbd61925ceaefeb57d3a1774d0ff06208e4bcf9 WatchSource:0}: Error finding container 70e53782a2377ebe858636dfabbd61925ceaefeb57d3a1774d0ff06208e4bcf9: Status 404 returned error can't find the container with id 70e53782a2377ebe858636dfabbd61925ceaefeb57d3a1774d0ff06208e4bcf9 Jul 08 18:17:48.075557 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.075402 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c228s" Jul 08 18:17:48.075875 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.075851 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cf354c_96c9_4feb_a464_d48b6a8d7fdf.slice/crio-cc1375a24cb885e37dbee5727cf54fdc0b51bb1b679dd43c6d296187f63f4eac WatchSource:0}: Error finding container cc1375a24cb885e37dbee5727cf54fdc0b51bb1b679dd43c6d296187f63f4eac: Status 404 returned error can't find the container with id cc1375a24cb885e37dbee5727cf54fdc0b51bb1b679dd43c6d296187f63f4eac Jul 08 18:17:48.082411 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.082259 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:17:48.086792 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.086765 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28adaf6_6f30_4afd_80cf_191bc1b23cc0.slice/crio-0a45d9610bceae9a469d8ce18f3de6cb7fd11e377c2bdac7bf905728f0addd38 WatchSource:0}: Error finding container 0a45d9610bceae9a469d8ce18f3de6cb7fd11e377c2bdac7bf905728f0addd38: Status 404 returned error can't find the container with id 0a45d9610bceae9a469d8ce18f3de6cb7fd11e377c2bdac7bf905728f0addd38 Jul 08 18:17:48.092449 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:17:48.092427 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5926e59d_0612_4f42_bc6c_d45f60b419c3.slice/crio-fae2c3794db1b758dce814f7ea6d3be283efc633e4fdb14b4a336abcafcb79a0 WatchSource:0}: Error finding container fae2c3794db1b758dce814f7ea6d3be283efc633e4fdb14b4a336abcafcb79a0: Status 404 returned error can't find the container with id fae2c3794db1b758dce814f7ea6d3be283efc633e4fdb14b4a336abcafcb79a0 Jul 08 18:17:48.327133 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.327051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:48.327284 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.327234 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:48.327348 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.327295 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:48.327348 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.327215 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:48.327348 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.327303 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:17:49.327281993 +0000 UTC m=+3.950869057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:48.327348 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.327348 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret podName:d38d4dce-6648-426f-98f7-d2fbaf4e89df nodeName:}" failed. No retries permitted until 2026-07-08 18:17:49.327337196 +0000 UTC m=+3.950924259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret") pod "global-pull-secret-syncer-v7kp9" (UID: "d38d4dce-6648-426f-98f7-d2fbaf4e89df") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:48.428334 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.428299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:48.428491 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.428440 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:17:48.428491 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.428464 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:17:48.428491 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.428478 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n9nhs for pod openshift-network-diagnostics/network-check-target-pfzrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:48.428698 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.428543 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs podName:a61af694-9ea9-4f7b-8509-cb463bb5d73b nodeName:}" failed. No retries permitted until 2026-07-08 18:17:49.428524525 +0000 UTC m=+4.052111602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nhs" (UniqueName: "kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs") pod "network-check-target-pfzrd" (UID: "a61af694-9ea9-4f7b-8509-cb463bb5d73b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:48.756521 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.755901 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:12:47 +0000 UTC" deadline="2028-04-04 14:39:01.187904986 +0000 UTC" Jul 08 18:17:48.756521 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.755967 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15260h21m12.431942131s" Jul 08 18:17:48.921592 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.921562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:48.921832 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.921713 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:17:48.922158 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.922133 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:48.922289 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:48.922237 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:17:48.933797 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.933762 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8lbkn" event={"ID":"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e","Type":"ContainerStarted","Data":"1046453f5bf3bb082c9e81cbfdad44435fcad4cea6fe567dfaa3d876bcbc24de"} Jul 08 18:17:48.937917 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.937862 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"fae2c3794db1b758dce814f7ea6d3be283efc633e4fdb14b4a336abcafcb79a0"} Jul 08 18:17:48.939610 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.939580 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-42rdp" event={"ID":"78707327-a0b1-4913-a4ba-6bf98aa3e78c","Type":"ContainerStarted","Data":"75f72121d368aab9f8b6af318d2fb782fcde36c055f4d7ca0e63a98df36997fc"} Jul 08 18:17:48.941144 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.941117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerStarted","Data":"8a4a10fd10d8b19f2e68c7e40aa1151bb9911c63b9c87caf3f59e6e2b6727551"} Jul 08 18:17:48.945212 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.945186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vflrw" event={"ID":"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59","Type":"ContainerStarted","Data":"316d9a23a62b1565d8cbf49a6b3aca3061185b473951e37ca7bf711ae7299b42"} Jul 08 18:17:48.948775 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.948751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c228s" event={"ID":"d28adaf6-6f30-4afd-80cf-191bc1b23cc0","Type":"ContainerStarted","Data":"0a45d9610bceae9a469d8ce18f3de6cb7fd11e377c2bdac7bf905728f0addd38"} Jul 08 18:17:48.951195 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.951168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vldb6" event={"ID":"f8cf354c-96c9-4feb-a464-d48b6a8d7fdf","Type":"ContainerStarted","Data":"cc1375a24cb885e37dbee5727cf54fdc0b51bb1b679dd43c6d296187f63f4eac"} Jul 08 18:17:48.953829 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.953805 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjk7q" event={"ID":"9e769af3-83dc-4841-95a4-b514d5e6b179","Type":"ContainerStarted","Data":"70e53782a2377ebe858636dfabbd61925ceaefeb57d3a1774d0ff06208e4bcf9"} Jul 08 18:17:48.955649 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:48.955611 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" event={"ID":"bfbd0e88-1340-470e-b342-c106adec2bb5","Type":"ContainerStarted","Data":"7598a25c3ec4526ea25a21dff0a328a0e202585b2ea4070d9253612690346cc7"} Jul 08 18:17:49.336731 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:49.336695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:49.336932 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:49.336765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:49.336993 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.336934 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:49.337049 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.337001 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:17:51.336980538 +0000 UTC m=+5.960567615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:49.337537 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.337448 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:49.337537 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.337507 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret podName:d38d4dce-6648-426f-98f7-d2fbaf4e89df nodeName:}" failed. No retries permitted until 2026-07-08 18:17:51.337489603 +0000 UTC m=+5.961076679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret") pod "global-pull-secret-syncer-v7kp9" (UID: "d38d4dce-6648-426f-98f7-d2fbaf4e89df") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:49.437988 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:49.437550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:49.437988 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.437778 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:17:49.437988 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.437804 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:17:49.437988 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.437817 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n9nhs for pod openshift-network-diagnostics/network-check-target-pfzrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:49.437988 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.437877 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs podName:a61af694-9ea9-4f7b-8509-cb463bb5d73b nodeName:}" failed. No retries permitted until 2026-07-08 18:17:51.437857763 +0000 UTC m=+6.061444838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nhs" (UniqueName: "kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs") pod "network-check-target-pfzrd" (UID: "a61af694-9ea9-4f7b-8509-cb463bb5d73b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:49.756491 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:49.756403 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:12:47 +0000 UTC" deadline="2028-02-01 01:28:13.027052671 +0000 UTC" Jul 08 18:17:49.756491 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:49.756445 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13735h10m23.270612846s" Jul 08 18:17:49.924288 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:49.924258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:49.924446 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:49.924402 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:17:50.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:50.921948 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:50.921995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:50.921973 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:50.922452 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:50.922076 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:17:50.922452 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:50.922247 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:17:51.353059 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:51.352977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:51.353213 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:51.353060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:51.353213 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.353182 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:51.353319 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.353255 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret podName:d38d4dce-6648-426f-98f7-d2fbaf4e89df nodeName:}" failed. No retries permitted until 2026-07-08 18:17:55.353237362 +0000 UTC m=+9.976824433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret") pod "global-pull-secret-syncer-v7kp9" (UID: "d38d4dce-6648-426f-98f7-d2fbaf4e89df") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:51.353647 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.353626 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:51.353734 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.353701 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:17:55.353685145 +0000 UTC m=+9.977272207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:51.454040 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:51.454006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:51.454225 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.454169 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:17:51.454225 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.454192 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:17:51.454225 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.454206 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n9nhs for pod openshift-network-diagnostics/network-check-target-pfzrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:51.454382 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.454260 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs podName:a61af694-9ea9-4f7b-8509-cb463bb5d73b nodeName:}" failed. No retries permitted until 2026-07-08 18:17:55.454241284 +0000 UTC m=+10.077828357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nhs" (UniqueName: "kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs") pod "network-check-target-pfzrd" (UID: "a61af694-9ea9-4f7b-8509-cb463bb5d73b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:51.922402 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:51.922367 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:51.922860 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:51.922508 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:17:52.922335 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:52.921848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:52.922335 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:52.921982 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:17:52.922852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:52.922395 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:52.922852 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:52.922501 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:17:53.922431 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:53.921593 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:53.922431 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:53.921993 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:17:54.921274 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:54.921243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:54.921441 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:54.921384 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:17:54.921503 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:54.921446 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:54.921589 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:54.921568 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:17:55.382507 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:55.382478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:55.382990 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:55.382531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:55.382990 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.382648 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:55.382990 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.382740 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:03.382709726 +0000 UTC m=+18.006296785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:17:55.382990 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.382649 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:55.382990 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.382800 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret podName:d38d4dce-6648-426f-98f7-d2fbaf4e89df nodeName:}" failed. No retries permitted until 2026-07-08 18:18:03.382784309 +0000 UTC m=+18.006371380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret") pod "global-pull-secret-syncer-v7kp9" (UID: "d38d4dce-6648-426f-98f7-d2fbaf4e89df") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:17:55.483697 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:55.483641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:55.483850 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.483808 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:17:55.483850 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.483834 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:17:55.483850 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.483847 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n9nhs for pod openshift-network-diagnostics/network-check-target-pfzrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:55.484002 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.483904 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs podName:a61af694-9ea9-4f7b-8509-cb463bb5d73b nodeName:}" failed. No retries permitted until 2026-07-08 18:18:03.483885885 +0000 UTC m=+18.107472948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nhs" (UniqueName: "kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs") pod "network-check-target-pfzrd" (UID: "a61af694-9ea9-4f7b-8509-cb463bb5d73b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:17:55.922239 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:55.922210 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:55.922411 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:55.922313 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:17:56.922319 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:56.922291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:56.922756 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:56.922299 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:56.922756 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:56.922405 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:17:56.922756 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:56.922518 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:17:57.921867 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:57.921779 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:57.922028 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:57.921924 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:17:58.922149 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:58.922118 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:17:58.922567 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:58.922126 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:17:58.922567 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:58.922241 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:17:58.922567 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:58.922330 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:17:59.921957 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:17:59.921922 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:17:59.922117 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:17:59.922048 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:00.921704 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:00.921671 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:00.922131 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:00.921788 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:00.922131 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:00.921858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:00.922131 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:00.921984 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:01.922078 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:01.922056 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:01.922392 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:01.922164 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:01.982108 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:01.982059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-42rdp" event={"ID":"78707327-a0b1-4913-a4ba-6bf98aa3e78c","Type":"ContainerStarted","Data":"84cb196543f6e13452c408fce226240f6190fc12f66186e4c8a1dc64c5edac8f"} Jul 08 18:18:01.983547 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:01.983527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" event={"ID":"a4bc3b05febacfc431990abc062b9833","Type":"ContainerStarted","Data":"64897633a1970c4ad513c3035712f5a0be3426ecac9d72531dc1040226c7d1b2"} Jul 08 18:18:02.011822 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:02.010624 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-42rdp" podStartSLOduration=2.481605142 podStartE2EDuration="16.010579687s" totalImagesPullingTime="13.528974545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:02.009780865 +0000 UTC m=+16.633367968" watchObservedRunningTime="2026-07-08 18:18:02.010579687 +0000 UTC m=+16.634166768" Jul 08 18:18:02.037402 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:02.037356 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-178.ec2.internal" podStartSLOduration=15.03733861 podStartE2EDuration="15.03733861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:02.037222766 +0000 UTC m=+16.660809847" watchObservedRunningTime="2026-07-08 18:18:02.03733861 +0000 UTC m=+16.660925690" Jul 08 18:18:02.922458 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:02.922260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:02.922906 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:02.922281 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:02.922906 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:02.922656 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:02.922906 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:02.922542 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:02.986647 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:02.986597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8lbkn" event={"ID":"8e202106-9ec8-4675-8bd9-f3c3ba45cb3e","Type":"ContainerStarted","Data":"0ddf788bdda6444df714c63a229d8941b1743472a629af021bde73b38e63f478"} Jul 08 18:18:03.010771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.010712 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8lbkn" podStartSLOduration=3.0515652 podStartE2EDuration="17.01069729s" totalImagesPullingTime="13.95913209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:03.010540776 +0000 UTC m=+17.634127855" watchObservedRunningTime="2026-07-08 18:18:03.01069729 +0000 UTC m=+17.634284369" Jul 08 18:18:03.447901 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.447871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:03.448050 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.447919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:03.448050 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.448018 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:18:03.448150 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.448071 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:19.448058137 +0000 UTC m=+34.071645207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:18:03.448150 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.448018 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:18:03.448239 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.448158 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret podName:d38d4dce-6648-426f-98f7-d2fbaf4e89df nodeName:}" failed. No retries permitted until 2026-07-08 18:18:19.448144723 +0000 UTC m=+34.071731792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret") pod "global-pull-secret-syncer-v7kp9" (UID: "d38d4dce-6648-426f-98f7-d2fbaf4e89df") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:18:03.549031 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.549004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:03.549187 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.549166 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:18:03.549244 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.549198 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:18:03.549244 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.549212 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n9nhs for pod openshift-network-diagnostics/network-check-target-pfzrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:18:03.549325 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.549262 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs podName:a61af694-9ea9-4f7b-8509-cb463bb5d73b nodeName:}" failed. No retries permitted until 2026-07-08 18:18:19.549246857 +0000 UTC m=+34.172833934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nhs" (UniqueName: "kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs") pod "network-check-target-pfzrd" (UID: "a61af694-9ea9-4f7b-8509-cb463bb5d73b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:18:03.921977 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.921942 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:03.922151 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:03.922058 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:03.993318 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.993282 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerStarted","Data":"2baca89c5b71d45f9c816d6dae9c613f3f9ad93d50c8691ba982528173d39010"} Jul 08 18:18:03.995025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.994989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vflrw" event={"ID":"c39b1b9d-ddb3-4f4e-96c9-67bba8530d59","Type":"ContainerStarted","Data":"f9216a68c5ef69a4c734d95d41124f269b3a3285ee66fd1d839362c9e2873749"} Jul 08 18:18:03.996643 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.996617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c228s" event={"ID":"d28adaf6-6f30-4afd-80cf-191bc1b23cc0","Type":"ContainerStarted","Data":"584a250413f776a0c62b68d78e679476f907e4ec5c9f11b898313a26f8fdb81b"} Jul 08 18:18:03.998074 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.998052 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vldb6" event={"ID":"f8cf354c-96c9-4feb-a464-d48b6a8d7fdf","Type":"ContainerStarted","Data":"a2391e40dc83b08aa321fd7ac03d00e83ace09b1065de2aa9c0d7e4cdc2c58cb"} Jul 08 18:18:03.999573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:03.999554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjk7q" event={"ID":"9e769af3-83dc-4841-95a4-b514d5e6b179","Type":"ContainerStarted","Data":"a1a1352479e6119e9e3b57d3d23ab81812cba1803cb6a306cfcb6aa53be02681"} Jul 08 18:18:04.001125 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:04.001099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" event={"ID":"bfbd0e88-1340-470e-b342-c106adec2bb5","Type":"ContainerStarted","Data":"80e896b11d6e7e55d09f04abf8d88bc8ad7a1db651e73e9455393a7ae23b1475"} Jul 08 18:18:04.002510 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:04.002489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" event={"ID":"3dd528dcab299f7ca40ff67e8a092082","Type":"ContainerStarted","Data":"493e7995bed8f632fd97d391fc43ebb769f9a3b48d9f86a36d3bdd2c13982ab5"} Jul 08 18:18:04.031250 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:04.031206 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bjk7q" podStartSLOduration=4.535997267 podStartE2EDuration="18.031192145s" totalImagesPullingTime="13.495194878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:04.030949768 +0000 UTC m=+18.654536845" watchObservedRunningTime="2026-07-08 18:18:04.031192145 +0000 UTC m=+18.654779224" Jul 08 18:18:04.045146 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:04.045110 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vldb6" podStartSLOduration=8.482471133 podStartE2EDuration="18.045098644s" totalImagesPullingTime="9.562627511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:04.04452664 +0000 UTC m=+18.668113720" watchObservedRunningTime="2026-07-08 18:18:04.045098644 +0000 UTC m=+18.668685722" Jul 08 18:18:04.063223 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:04.063191 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c228s" podStartSLOduration=8.488738962 podStartE2EDuration="18.063181566s" totalImagesPullingTime="9.574442604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:04.062860674 +0000 UTC m=+18.686447753" watchObservedRunningTime="2026-07-08 18:18:04.063181566 +0000 UTC m=+18.686768644" Jul 08 18:18:04.091864 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:04.091832 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vflrw" podStartSLOduration=4.525533249 podStartE2EDuration="18.091822828s" totalImagesPullingTime="13.566289579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:04.091704014 +0000 UTC m=+18.715291095" watchObservedRunningTime="2026-07-08 18:18:04.091822828 +0000 UTC m=+18.715409908" Jul 08 18:18:04.921524 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:04.921494 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:04.921718 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:04.921537 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:04.921718 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:04.921605 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:04.921840 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:04.921762 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:05.922212 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:05.922181 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:05.922764 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:05.922289 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:06.922229 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:06.922121 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:06.922537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:06.922121 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:06.922537 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:06.922335 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:06.922537 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:06.922420 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:07.009783 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.009738 2565 generic.go:356] "Generic (PLEG): container finished" podID="3dd528dcab299f7ca40ff67e8a092082" containerID="493e7995bed8f632fd97d391fc43ebb769f9a3b48d9f86a36d3bdd2c13982ab5" exitCode=0 Jul 08 18:18:07.009913 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.009785 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" event={"ID":"3dd528dcab299f7ca40ff67e8a092082","Type":"ContainerDied","Data":"493e7995bed8f632fd97d391fc43ebb769f9a3b48d9f86a36d3bdd2c13982ab5"} Jul 08 18:18:07.012112 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.012085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"9f988933a9d4fc9d5c51c046c162d8ad0b566c160cbdcc27128f1d6b9d18a5d5"} Jul 08 18:18:07.012213 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.012165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"51bcd77b943a931f3f96a53745ea561bc0263330bc5c14ab6aaefc4545866797"} Jul 08 18:18:07.013812 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.013791 2565 generic.go:356] "Generic (PLEG): container finished" podID="e7d23d0c-cd9d-43bf-b2bb-569e389eae4b" containerID="2baca89c5b71d45f9c816d6dae9c613f3f9ad93d50c8691ba982528173d39010" exitCode=0 Jul 08 18:18:07.013910 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.013826 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerDied","Data":"2baca89c5b71d45f9c816d6dae9c613f3f9ad93d50c8691ba982528173d39010"} Jul 08 18:18:07.326250 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.326122 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 18:18:07.770072 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.769967 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T18:18:07.326248491Z","UUID":"dcbb01d6-0502-4ec3-8476-c258b4d20299","Handler":null,"Name":"","Endpoint":""} Jul 08 18:18:07.771555 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.771522 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 18:18:07.771555 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.771560 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 18:18:07.921375 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:07.921343 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:07.921582 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:07.921477 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:08.018530 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.018499 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" event={"ID":"3dd528dcab299f7ca40ff67e8a092082","Type":"ContainerStarted","Data":"bf75725695f7b3d3044e86f0caadac604090f584ffc0cb2f21600bb2ec8b60b6"} Jul 08 18:18:08.021796 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.021736 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:18:08.022127 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.022106 2565 generic.go:356] "Generic (PLEG): container finished" podID="5926e59d-0612-4f42-bc6c-d45f60b419c3" containerID="9f988933a9d4fc9d5c51c046c162d8ad0b566c160cbdcc27128f1d6b9d18a5d5" exitCode=1 Jul 08 18:18:08.022216 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.022172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerDied","Data":"9f988933a9d4fc9d5c51c046c162d8ad0b566c160cbdcc27128f1d6b9d18a5d5"} Jul 08 18:18:08.022273 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.022254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"d8c8d9897a862dfdcdfaeaf85fa0940350f4353170bd057f87dfa76ffdd42a1d"} Jul 08 18:18:08.022326 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.022303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"5202c422ca4ee9f792ae45bb8b7a32c51ca8e7fd7d9fe5bb943565a056f14087"} Jul 08 18:18:08.022376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.022348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"462f39e4ecd382c0a1ab1dab24c1e14da5741b44eff044b4a4057f4154c05722"} Jul 08 18:18:08.022427 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.022396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"be7c11370417d7a51bd91a5b814da4f707c31d78a921a1890804fbcc6ee846a7"} Jul 08 18:18:08.024879 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.024851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" event={"ID":"bfbd0e88-1340-470e-b342-c106adec2bb5","Type":"ContainerStarted","Data":"98e8da148e136a947ad1910585dabcbd34f6e32c0612da336dd474547bbf2c9d"} Jul 08 18:18:08.036103 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.036064 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-178.ec2.internal" podStartSLOduration=21.03605249 podStartE2EDuration="21.03605249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:08.035456385 +0000 UTC m=+22.659043463" watchObservedRunningTime="2026-07-08 18:18:08.03605249 +0000 UTC m=+22.659639568" Jul 08 18:18:08.344427 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.344402 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:18:08.345001 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.344985 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:18:08.922008 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.921982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:08.922225 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:08.921982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:08.922225 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:08.922073 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:08.922225 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:08.922140 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:09.028472 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:09.028447 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" event={"ID":"bfbd0e88-1340-470e-b342-c106adec2bb5","Type":"ContainerStarted","Data":"f00b504e1e9d6fd203b2fe9973fbb079dfde7f26be97b774c18378af4faed4a4"} Jul 08 18:18:09.029025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:09.028641 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:18:09.031485 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:09.029696 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vldb6" Jul 08 18:18:09.047078 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:09.047039 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmcwm" podStartSLOduration=4.433703267 podStartE2EDuration="23.047029083s" totalImagesPullingTime="18.613325816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:09.046656458 +0000 UTC m=+23.670243538" watchObservedRunningTime="2026-07-08 18:18:09.047029083 +0000 UTC m=+23.670616162" Jul 08 18:18:09.921998 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:09.921968 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:09.922173 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:09.922102 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:10.032963 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:10.032939 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:18:10.033397 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:10.033359 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"b206d8153b4cb832fcfa340c81454b8e2aafff4d25755ed16870e99eaf4e3b8e"} Jul 08 18:18:10.921949 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:10.921921 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:10.922133 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:10.922022 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:10.922133 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:10.922074 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:10.922250 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:10.922151 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:11.921873 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:11.921853 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:11.922202 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:11.921949 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:12.042076 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.041901 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:18:12.042461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.042434 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"a587599a5722fdefe94dd818fa28bddcb3caf558bb0953f68aa446105023f2d4"} Jul 08 18:18:12.042799 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.042777 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:18:12.042892 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.042855 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:18:12.042945 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.042909 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:18:12.042988 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.042967 2565 scope.go:117] "RemoveContainer" containerID="9f988933a9d4fc9d5c51c046c162d8ad0b566c160cbdcc27128f1d6b9d18a5d5" Jul 08 18:18:12.058619 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.058600 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:18:12.059784 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.059768 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:18:12.921839 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.921805 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:12.921940 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:12.921920 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:12.922209 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:12.921805 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:12.922209 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:12.922068 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:13.052326 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.052307 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:18:13.052740 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.052716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" event={"ID":"5926e59d-0612-4f42-bc6c-d45f60b419c3","Type":"ContainerStarted","Data":"282ebcd901fdf05270a17864718873c268bae873e11c81e29e9e991f174efe71"} Jul 08 18:18:13.054776 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.054750 2565 generic.go:356] "Generic (PLEG): container finished" podID="e7d23d0c-cd9d-43bf-b2bb-569e389eae4b" containerID="f7edb4dab97b1de74a578a04d253950c90dccec97e5a0a12ad9c108e2171d8f1" exitCode=0 Jul 08 18:18:13.054881 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.054791 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerDied","Data":"f7edb4dab97b1de74a578a04d253950c90dccec97e5a0a12ad9c108e2171d8f1"} Jul 08 18:18:13.083820 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.083774 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" podStartSLOduration=8.49055955 podStartE2EDuration="27.083757236s" totalImagesPullingTime="18.593197686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:13.081876841 +0000 UTC m=+27.705463954" watchObservedRunningTime="2026-07-08 18:18:13.083757236 +0000 UTC m=+27.707344306" Jul 08 18:18:13.844162 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.844139 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7kp9"] Jul 08 18:18:13.844433 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.844420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:13.844521 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:13.844502 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:13.846507 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.846476 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pfzrd"] Jul 08 18:18:13.846590 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.846579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:13.846672 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:13.846647 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:13.857602 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.857578 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4dl6h"] Jul 08 18:18:13.857733 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:13.857718 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:13.857871 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:13.857832 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:14.058640 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:14.058578 2565 generic.go:356] "Generic (PLEG): container finished" podID="e7d23d0c-cd9d-43bf-b2bb-569e389eae4b" containerID="249972b78eb22baf30343068987ee396405673aaf2e1184835d9dd72f04135a6" exitCode=0 Jul 08 18:18:14.058983 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:14.058673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerDied","Data":"249972b78eb22baf30343068987ee396405673aaf2e1184835d9dd72f04135a6"} Jul 08 18:18:15.063225 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:15.063200 2565 generic.go:356] "Generic (PLEG): container finished" podID="e7d23d0c-cd9d-43bf-b2bb-569e389eae4b" containerID="95868b0291995e2101b9eb8b5ba6eaa81be719c28ffe68f9cc55e229f8485430" exitCode=0 Jul 08 18:18:15.063489 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:15.063246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerDied","Data":"95868b0291995e2101b9eb8b5ba6eaa81be719c28ffe68f9cc55e229f8485430"} Jul 08 18:18:15.923288 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:15.923127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:15.923408 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:15.923387 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:15.923475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:15.923166 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:15.923475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:15.923197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:15.923566 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:15.923497 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:15.923617 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:15.923568 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:17.921842 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:17.921817 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:17.921842 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:17.921836 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:17.922332 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:17.921911 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7kp9" podUID="d38d4dce-6648-426f-98f7-d2fbaf4e89df" Jul 08 18:18:17.922332 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:17.921966 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:18:17.922332 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:17.922006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:17.922332 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:17.922070 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfzrd" podUID="a61af694-9ea9-4f7b-8509-cb463bb5d73b" Jul 08 18:18:19.469224 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.469190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:19.469779 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.469235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:19.469779 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.469327 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:18:19.469779 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.469335 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:18:19.469779 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.469373 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:51.469360224 +0000 UTC m=+66.092947294 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:18:19.469779 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.469385 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret podName:d38d4dce-6648-426f-98f7-d2fbaf4e89df nodeName:}" failed. No retries permitted until 2026-07-08 18:18:51.469379413 +0000 UTC m=+66.092966474 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret") pod "global-pull-secret-syncer-v7kp9" (UID: "d38d4dce-6648-426f-98f7-d2fbaf4e89df") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:18:19.569726 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.569682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:19.569878 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.569825 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:18:19.569878 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.569839 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:18:19.569878 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.569848 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n9nhs for pod openshift-network-diagnostics/network-check-target-pfzrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:18:19.570027 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.569895 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs podName:a61af694-9ea9-4f7b-8509-cb463bb5d73b nodeName:}" failed. No retries permitted until 2026-07-08 18:18:51.569879741 +0000 UTC m=+66.193466808 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nhs" (UniqueName: "kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs") pod "network-check-target-pfzrd" (UID: "a61af694-9ea9-4f7b-8509-cb463bb5d73b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:18:19.625673 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.625641 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-178.ec2.internal" event="NodeReady" Jul 08 18:18:19.625766 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.625759 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 18:18:19.660416 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.660394 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj"] Jul 08 18:18:19.676578 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.676555 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c454f47b9-658l2"] Jul 08 18:18:19.676733 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.676714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj" Jul 08 18:18:19.681031 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.680134 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:18:19.681031 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.680352 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pcncz\"" Jul 08 18:18:19.681031 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.680591 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 18:18:19.691045 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.691027 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv"] Jul 08 18:18:19.691156 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.691143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.693596 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.693486 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hm9k9\"" Jul 08 18:18:19.693596 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.693517 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 18:18:19.693596 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.693554 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 18:18:19.693821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.693607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 18:18:19.700034 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.700014 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 18:18:19.702957 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.702939 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-g44vm"] Jul 08 18:18:19.703102 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.703082 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:19.705332 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.705315 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 18:18:19.705332 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.705328 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-n4nnk\"" Jul 08 18:18:19.705604 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.705588 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 18:18:19.716997 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.716980 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g"] Jul 08 18:18:19.717109 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.717092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.720091 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.720042 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:18:19.720091 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.720063 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 18:18:19.720236 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.720094 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 18:18:19.720236 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.720124 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-psjps\"" Jul 08 18:18:19.720236 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.720134 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 18:18:19.728880 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.728861 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 18:18:19.729171 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.729156 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj"] Jul 08 18:18:19.729227 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.729180 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv"] Jul 08 18:18:19.729227 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.729198 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g"] Jul 08 18:18:19.729227 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.729219 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dn82h"] Jul 08 18:18:19.729345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.729268 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g" Jul 08 18:18:19.731493 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.731477 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 18:18:19.731583 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.731505 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 18:18:19.731646 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.731560 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-87fdv\"" Jul 08 18:18:19.749208 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.749189 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5d65v"] Jul 08 18:18:19.749315 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.749297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.751810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.751792 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 18:18:19.751901 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.751825 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 18:18:19.751901 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.751792 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4sdr\"" Jul 08 18:18:19.761326 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.761311 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c454f47b9-658l2"] Jul 08 18:18:19.761418 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.761334 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-g44vm"] Jul 08 18:18:19.761418 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.761348 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dn82h"] Jul 08 18:18:19.761418 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.761363 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5d65v"] Jul 08 18:18:19.761418 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.761409 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5d65v" Jul 08 18:18:19.763861 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.763839 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 18:18:19.763939 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.763877 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 18:18:19.764016 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.763847 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 18:18:19.764095 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.764060 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r6bqm\"" Jul 08 18:18:19.770299 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.770281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54blg\" (UniqueName: \"kubernetes.io/projected/45e06774-ee41-48b8-9f76-420e92a9dd62-kube-api-access-54blg\") pod \"volume-data-source-validator-5d456fdbcd-4dbqj\" (UID: \"45e06774-ee41-48b8-9f76-420e92a9dd62\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj" Jul 08 18:18:19.850198 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.850162 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vldb6_f8cf354c-96c9-4feb-a464-d48b6a8d7fdf/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:19.871081 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:19.871177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54blg\" (UniqueName: \"kubernetes.io/projected/45e06774-ee41-48b8-9f76-420e92a9dd62-kube-api-access-54blg\") pod \"volume-data-source-validator-5d456fdbcd-4dbqj\" (UID: \"45e06774-ee41-48b8-9f76-420e92a9dd62\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj" Jul 08 18:18:19.871177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871134 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a6e10b2d-d75a-41ba-993e-35bfe3217286-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:19.871177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73787897-ffc8-47cb-befc-1d8af360d229-tmp-dir\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.871345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbjsh\" (UniqueName: \"kubernetes.io/projected/e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4-kube-api-access-wbjsh\") pod \"ingress-canary-5d65v\" (UID: \"e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4\") " pod="openshift-ingress-canary/ingress-canary-5d65v" Jul 08 18:18:19.871345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac2c03c-dff8-4df8-8400-0e30266cabfa-trusted-ca\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.871345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09e31dfd-bf6f-446a-af68-bc4c01a1e861-ca-trust-extracted\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.871345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ss2\" (UniqueName: \"kubernetes.io/projected/fac2c03c-dff8-4df8-8400-0e30266cabfa-kube-api-access-88ss2\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.871345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-certificates\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.871573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871355 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-installation-pull-secrets\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.871573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4-cert\") pod \"ingress-canary-5d65v\" (UID: \"e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4\") " pod="openshift-ingress-canary/ingress-canary-5d65v" Jul 08 18:18:19.871573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871417 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-bound-sa-token\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.871573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.871573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-image-registry-private-configuration\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.871573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-trusted-ca\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.871793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4bc\" (UniqueName: \"kubernetes.io/projected/73787897-ffc8-47cb-befc-1d8af360d229-kube-api-access-zg4bc\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.871793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fac2c03c-dff8-4df8-8400-0e30266cabfa-serving-cert\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.871793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/73787897-ffc8-47cb-befc-1d8af360d229-config-volume\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.871793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7g2n\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-kube-api-access-x7g2n\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.871793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fac2c03c-dff8-4df8-8400-0e30266cabfa-config\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.871793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.871987 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.871814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzzf\" (UniqueName: \"kubernetes.io/projected/54c16e39-44bf-492f-8c71-384ea7e89be6-kube-api-access-slzzf\") pod \"network-check-source-989f8b8dc-26r5g\" (UID: \"54c16e39-44bf-492f-8c71-384ea7e89be6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g" Jul 08 18:18:19.881278 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.881254 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54blg\" (UniqueName: \"kubernetes.io/projected/45e06774-ee41-48b8-9f76-420e92a9dd62-kube-api-access-54blg\") pod \"volume-data-source-validator-5d456fdbcd-4dbqj\" (UID: \"45e06774-ee41-48b8-9f76-420e92a9dd62\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj" Jul 08 18:18:19.922265 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.922243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:19.922370 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.922243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:19.922427 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.922412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:19.925365 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.925343 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 18:18:19.925475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.925402 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 18:18:19.925475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.925351 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kcpkw\"" Jul 08 18:18:19.925592 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.925555 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9qsc4\"" Jul 08 18:18:19.972984 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.972921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fac2c03c-dff8-4df8-8400-0e30266cabfa-serving-cert\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.972984 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.972958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/73787897-ffc8-47cb-befc-1d8af360d229-config-volume\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.973130 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.972999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7g2n\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-kube-api-access-x7g2n\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.973130 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fac2c03c-dff8-4df8-8400-0e30266cabfa-config\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.973130 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.973277 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.973147 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:18:19.973277 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.973158 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:18:19.973277 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzzf\" (UniqueName: \"kubernetes.io/projected/54c16e39-44bf-492f-8c71-384ea7e89be6-kube-api-access-slzzf\") pod \"network-check-source-989f8b8dc-26r5g\" (UID: \"54c16e39-44bf-492f-8c71-384ea7e89be6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g" Jul 08 18:18:19.973277 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.973208 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:20.473191451 +0000 UTC m=+35.096778513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:18:19.973277 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973265 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973313 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a6e10b2d-d75a-41ba-993e-35bfe3217286-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73787897-ffc8-47cb-befc-1d8af360d229-tmp-dir\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbjsh\" (UniqueName: \"kubernetes.io/projected/e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4-kube-api-access-wbjsh\") pod \"ingress-canary-5d65v\" (UID: \"e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4\") " pod="openshift-ingress-canary/ingress-canary-5d65v" Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac2c03c-dff8-4df8-8400-0e30266cabfa-trusted-ca\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09e31dfd-bf6f-446a-af68-bc4c01a1e861-ca-trust-extracted\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.973445 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ss2\" (UniqueName: \"kubernetes.io/projected/fac2c03c-dff8-4df8-8400-0e30266cabfa-kube-api-access-88ss2\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973500 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-certificates\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.973523 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.973514 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:20.473494728 +0000 UTC m=+35.097081799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:18:19.974014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-installation-pull-secrets\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.974014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4-cert\") pod \"ingress-canary-5d65v\" (UID: \"e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4\") " pod="openshift-ingress-canary/ingress-canary-5d65v" Jul 08 18:18:19.974014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-bound-sa-token\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.974014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.974014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973687 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73787897-ffc8-47cb-befc-1d8af360d229-tmp-dir\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.974014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-image-registry-private-configuration\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.974014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-trusted-ca\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.974014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4bc\" (UniqueName: \"kubernetes.io/projected/73787897-ffc8-47cb-befc-1d8af360d229-kube-api-access-zg4bc\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.974390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.974015 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fac2c03c-dff8-4df8-8400-0e30266cabfa-config\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.974390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.974021 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09e31dfd-bf6f-446a-af68-bc4c01a1e861-ca-trust-extracted\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.974390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.974044 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a6e10b2d-d75a-41ba-993e-35bfe3217286-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:19.974390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.974116 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-certificates\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.974390 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.974210 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:18:19.974390 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:19.974274 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:20.474256235 +0000 UTC m=+35.097843306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:18:19.974390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.973643 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/73787897-ffc8-47cb-befc-1d8af360d229-config-volume\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.974755 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.974677 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-trusted-ca\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.974803 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.974761 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac2c03c-dff8-4df8-8400-0e30266cabfa-trusted-ca\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.976971 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.976919 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-image-registry-private-configuration\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.976971 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.976968 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fac2c03c-dff8-4df8-8400-0e30266cabfa-serving-cert\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.977115 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.977014 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-installation-pull-secrets\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.977385 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.977356 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4-cert\") pod \"ingress-canary-5d65v\" (UID: \"e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4\") " pod="openshift-ingress-canary/ingress-canary-5d65v" Jul 08 18:18:19.983366 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.983321 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzzf\" (UniqueName: \"kubernetes.io/projected/54c16e39-44bf-492f-8c71-384ea7e89be6-kube-api-access-slzzf\") pod \"network-check-source-989f8b8dc-26r5g\" (UID: \"54c16e39-44bf-492f-8c71-384ea7e89be6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g" Jul 08 18:18:19.983905 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.983719 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ss2\" (UniqueName: \"kubernetes.io/projected/fac2c03c-dff8-4df8-8400-0e30266cabfa-kube-api-access-88ss2\") pod \"console-operator-68b757fb67-g44vm\" (UID: \"fac2c03c-dff8-4df8-8400-0e30266cabfa\") " pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:19.983905 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.983862 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4bc\" (UniqueName: \"kubernetes.io/projected/73787897-ffc8-47cb-befc-1d8af360d229-kube-api-access-zg4bc\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:19.984487 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.984452 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7g2n\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-kube-api-access-x7g2n\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.984743 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.984722 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-bound-sa-token\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:19.984947 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.984917 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbjsh\" (UniqueName: \"kubernetes.io/projected/e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4-kube-api-access-wbjsh\") pod \"ingress-canary-5d65v\" (UID: \"e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4\") " pod="openshift-ingress-canary/ingress-canary-5d65v" Jul 08 18:18:19.990049 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:19.990028 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj" Jul 08 18:18:20.027514 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.027488 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:20.030114 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.030070 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vldb6_f8cf354c-96c9-4feb-a464-d48b6a8d7fdf/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:20.038272 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.038250 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g" Jul 08 18:18:20.070250 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.070230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5d65v" Jul 08 18:18:20.145866 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.145838 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vldb6_f8cf354c-96c9-4feb-a464-d48b6a8d7fdf/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:20.246267 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.246228 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vldb6_f8cf354c-96c9-4feb-a464-d48b6a8d7fdf/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:20.376324 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.376292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vldb6_f8cf354c-96c9-4feb-a464-d48b6a8d7fdf/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:20.435355 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.435301 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vldb6_f8cf354c-96c9-4feb-a464-d48b6a8d7fdf/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:20.477930 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.477906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.477971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:20.478021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:20.478084 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:20.478130 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:20.478138 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:20.478144 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:20.478167 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:21.478145885 +0000 UTC m=+36.101732965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:20.478188 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:21.478174916 +0000 UTC m=+36.101761978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:18:20.478481 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:20.478207 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:21.478196079 +0000 UTC m=+36.101783136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:18:21.410802 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:21.410689 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5d65v"] Jul 08 18:18:21.423378 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:21.423352 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-g44vm"] Jul 08 18:18:21.438312 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:18:21.438283 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ac3f0b_d6c5_4d56_ad27_e39d04727ed4.slice/crio-19c5d4a98b5b666de443e4c0a7cf82dd62517539648c176fd7e1cb4fd2c79e2f WatchSource:0}: Error finding container 19c5d4a98b5b666de443e4c0a7cf82dd62517539648c176fd7e1cb4fd2c79e2f: Status 404 returned error can't find the container with id 19c5d4a98b5b666de443e4c0a7cf82dd62517539648c176fd7e1cb4fd2c79e2f Jul 08 18:18:21.442124 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:18:21.442101 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac2c03c_dff8_4df8_8400_0e30266cabfa.slice/crio-69ba3cc0e4e21a137b0f89f0e8dc64acedf12d7ad36f490398816e9f54e39887 WatchSource:0}: Error finding container 69ba3cc0e4e21a137b0f89f0e8dc64acedf12d7ad36f490398816e9f54e39887: Status 404 returned error can't find the container with id 69ba3cc0e4e21a137b0f89f0e8dc64acedf12d7ad36f490398816e9f54e39887 Jul 08 18:18:21.487022 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:21.486997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:21.487060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:21.487116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:21.487138 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:21.487189 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:23.487171485 +0000 UTC m=+38.110758561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:21.487226 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:21.487246 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:21.487200 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:21.487308 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:23.48729075 +0000 UTC m=+38.110877825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:18:21.487364 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:21.487364 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:23.487345543 +0000 UTC m=+38.110932608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:18:21.641350 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:21.641323 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj"] Jul 08 18:18:21.644325 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:21.644296 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g"] Jul 08 18:18:21.647033 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:18:21.647009 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c16e39_44bf_492f_8c71_384ea7e89be6.slice/crio-e2b18cb87439778339fe57b0a98eb2656f654055b88b7508d5f4148b8b1c504b WatchSource:0}: Error finding container e2b18cb87439778339fe57b0a98eb2656f654055b88b7508d5f4148b8b1c504b: Status 404 returned error can't find the container with id e2b18cb87439778339fe57b0a98eb2656f654055b88b7508d5f4148b8b1c504b Jul 08 18:18:21.648153 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:18:21.648121 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e06774_ee41_48b8_9f76_420e92a9dd62.slice/crio-d1d49d29f21bebc125319aafc2bda61fb1d76dff3ead67869a947c24a96c5bfb WatchSource:0}: Error finding container d1d49d29f21bebc125319aafc2bda61fb1d76dff3ead67869a947c24a96c5bfb: Status 404 returned error can't find the container with id d1d49d29f21bebc125319aafc2bda61fb1d76dff3ead67869a947c24a96c5bfb Jul 08 18:18:22.082019 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:22.081977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g" event={"ID":"54c16e39-44bf-492f-8c71-384ea7e89be6","Type":"ContainerStarted","Data":"e2b18cb87439778339fe57b0a98eb2656f654055b88b7508d5f4148b8b1c504b"} Jul 08 18:18:22.083383 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:22.083348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj" event={"ID":"45e06774-ee41-48b8-9f76-420e92a9dd62","Type":"ContainerStarted","Data":"d1d49d29f21bebc125319aafc2bda61fb1d76dff3ead67869a947c24a96c5bfb"} Jul 08 18:18:22.084413 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:22.084389 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerStarted","Data":"69ba3cc0e4e21a137b0f89f0e8dc64acedf12d7ad36f490398816e9f54e39887"} Jul 08 18:18:22.086834 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:22.086814 2565 generic.go:356] "Generic (PLEG): container finished" podID="e7d23d0c-cd9d-43bf-b2bb-569e389eae4b" containerID="c899c4d04607808f680c394349339ad377b9a20ee4d90d5d64c97c8028e1cf93" exitCode=0 Jul 08 18:18:22.086926 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:22.086892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerDied","Data":"c899c4d04607808f680c394349339ad377b9a20ee4d90d5d64c97c8028e1cf93"} Jul 08 18:18:22.087935 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:22.087917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5d65v" event={"ID":"e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4","Type":"ContainerStarted","Data":"19c5d4a98b5b666de443e4c0a7cf82dd62517539648c176fd7e1cb4fd2c79e2f"} Jul 08 18:18:23.095118 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:23.094875 2565 generic.go:356] "Generic (PLEG): container finished" podID="e7d23d0c-cd9d-43bf-b2bb-569e389eae4b" containerID="a222350c78792ab1e3f60e0b3bb9708b3afeee2e35f05d748956ccefe0bb5bbc" exitCode=0 Jul 08 18:18:23.095118 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:23.094918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerDied","Data":"a222350c78792ab1e3f60e0b3bb9708b3afeee2e35f05d748956ccefe0bb5bbc"} Jul 08 18:18:23.503715 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:23.503687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:23.503822 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:23.503766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:23.503885 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:23.503820 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:18:23.503885 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:23.503840 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:18:23.503974 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:23.503837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:23.503974 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:23.503897 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:27.503877006 +0000 UTC m=+42.127464079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:18:23.503974 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:23.503921 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:18:23.503974 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:23.503942 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:18:23.504159 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:23.503975 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:27.503956797 +0000 UTC m=+42.127543858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:18:23.504159 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:23.503997 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:27.503987375 +0000 UTC m=+42.127574435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:18:24.101135 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:24.101096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" event={"ID":"e7d23d0c-cd9d-43bf-b2bb-569e389eae4b","Type":"ContainerStarted","Data":"64570682c93a9898b625aa2b7bd72b2b608fca3595f6b8579d8cf243b3d0677d"} Jul 08 18:18:24.102796 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:24.102772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj" event={"ID":"45e06774-ee41-48b8-9f76-420e92a9dd62","Type":"ContainerStarted","Data":"6fb424e623123732c4035a791368a6074aca0a5d63387fd0feeff384f6357701"} Jul 08 18:18:24.130881 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:24.130830 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dzfqj" podStartSLOduration=11.78088592 podStartE2EDuration="38.13081597s" totalImagesPullingTime="26.34993005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:24.12923966 +0000 UTC m=+38.752826778" watchObservedRunningTime="2026-07-08 18:18:24.13081597 +0000 UTC m=+38.754403049" Jul 08 18:18:24.145102 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:24.144893 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4dbqj" podStartSLOduration=34.324033124 podStartE2EDuration="36.144881272s" totalImagesPullingTime="1.820848148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:24.1446357 +0000 UTC m=+38.768222783" watchObservedRunningTime="2026-07-08 18:18:24.144881272 +0000 UTC m=+38.768468352" Jul 08 18:18:26.110274 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:26.110236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5d65v" event={"ID":"e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4","Type":"ContainerStarted","Data":"f5d39a673e04f6e11b80f2f707841ce5d0e48db08bc92c246d30f2da088b458e"} Jul 08 18:18:26.112227 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:26.112204 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/0.log" Jul 08 18:18:26.112360 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:26.112250 2565 generic.go:356] "Generic (PLEG): container finished" podID="fac2c03c-dff8-4df8-8400-0e30266cabfa" containerID="c69b07b6f8e47555c7d7a160f38102a529c0d3f22d5d3134470ad5a3ac352d23" exitCode=255 Jul 08 18:18:26.112360 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:26.112288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerDied","Data":"c69b07b6f8e47555c7d7a160f38102a529c0d3f22d5d3134470ad5a3ac352d23"} Jul 08 18:18:26.112558 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:26.112543 2565 scope.go:117] "RemoveContainer" containerID="c69b07b6f8e47555c7d7a160f38102a529c0d3f22d5d3134470ad5a3ac352d23" Jul 08 18:18:26.128800 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:26.128764 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5d65v" podStartSLOduration=3.298071039 podStartE2EDuration="7.128746647s" totalImagesPullingTime="3.830675608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:26.126920338 +0000 UTC m=+40.750507417" watchObservedRunningTime="2026-07-08 18:18:26.128746647 +0000 UTC m=+40.752333728" Jul 08 18:18:27.116674 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:27.116641 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/0.log" Jul 08 18:18:27.116959 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:27.116711 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerStarted","Data":"ff8f82708b05b50d9821dbcd04172e7d594f5cab3ae44d8a659a1d81af1cd8f6"} Jul 08 18:18:27.534787 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:27.534761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:27.534921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:27.534815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:27.534921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:27.534853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:27.535011 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:27.534923 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:18:27.535011 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:27.534966 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:18:27.535011 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:27.534981 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:18:27.535011 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:27.534994 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:35.534973342 +0000 UTC m=+50.158560409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:18:27.535172 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:27.535017 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:35.535005747 +0000 UTC m=+50.158592824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:18:27.535172 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:27.534928 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:18:27.535172 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:27.535088 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:35.535069524 +0000 UTC m=+50.158656610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:18:28.121720 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:28.121697 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/1.log" Jul 08 18:18:28.122100 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:28.122086 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/0.log" Jul 08 18:18:28.122159 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:28.122117 2565 generic.go:356] "Generic (PLEG): container finished" podID="fac2c03c-dff8-4df8-8400-0e30266cabfa" containerID="ff8f82708b05b50d9821dbcd04172e7d594f5cab3ae44d8a659a1d81af1cd8f6" exitCode=255 Jul 08 18:18:28.122218 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:28.122203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerDied","Data":"ff8f82708b05b50d9821dbcd04172e7d594f5cab3ae44d8a659a1d81af1cd8f6"} Jul 08 18:18:28.122271 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:28.122249 2565 scope.go:117] "RemoveContainer" containerID="c69b07b6f8e47555c7d7a160f38102a529c0d3f22d5d3134470ad5a3ac352d23" Jul 08 18:18:28.122460 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:28.122439 2565 scope.go:117] "RemoveContainer" containerID="ff8f82708b05b50d9821dbcd04172e7d594f5cab3ae44d8a659a1d81af1cd8f6" Jul 08 18:18:28.122618 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:28.122598 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:18:28.123821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:28.123804 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g" event={"ID":"54c16e39-44bf-492f-8c71-384ea7e89be6","Type":"ContainerStarted","Data":"e129c9e0e24db1b22c9238bbd615f7b088796d50580866075916cd14bbcd0dad"} Jul 08 18:18:28.161545 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:28.161506 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-26r5g" podStartSLOduration=31.374386433 podStartE2EDuration="37.161489061s" totalImagesPullingTime="5.787102628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:28.160754793 +0000 UTC m=+42.784341885" watchObservedRunningTime="2026-07-08 18:18:28.161489061 +0000 UTC m=+42.785076137" Jul 08 18:18:29.128142 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:29.128114 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/1.log" Jul 08 18:18:29.128566 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:29.128427 2565 scope.go:117] "RemoveContainer" containerID="ff8f82708b05b50d9821dbcd04172e7d594f5cab3ae44d8a659a1d81af1cd8f6" Jul 08 18:18:29.128566 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:29.128558 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:18:30.028582 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:30.028557 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:30.028797 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:30.028599 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:30.130892 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:30.130861 2565 scope.go:117] "RemoveContainer" containerID="ff8f82708b05b50d9821dbcd04172e7d594f5cab3ae44d8a659a1d81af1cd8f6" Jul 08 18:18:30.131190 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:30.131003 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:18:35.590643 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:35.590605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:35.590673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:35.590721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:35.590768 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:35.590777 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:35.590800 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:35.590805 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:35.590832 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:51.590816277 +0000 UTC m=+66.214403335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:35.590854 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:51.59083792 +0000 UTC m=+66.214424991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:18:35.591156 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:35.590874 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:18:51.590865303 +0000 UTC m=+66.214452362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:18:42.922567 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:42.922536 2565 scope.go:117] "RemoveContainer" containerID="ff8f82708b05b50d9821dbcd04172e7d594f5cab3ae44d8a659a1d81af1cd8f6" Jul 08 18:18:43.163752 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:43.163730 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/2.log" Jul 08 18:18:43.164145 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:43.164131 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/1.log" Jul 08 18:18:43.164208 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:43.164162 2565 generic.go:356] "Generic (PLEG): container finished" podID="fac2c03c-dff8-4df8-8400-0e30266cabfa" containerID="e3933aa98b22e82394e3bf58a9e85f56e501c45bbfaf1f467074dd646be1bebb" exitCode=255 Jul 08 18:18:43.164251 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:43.164224 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerDied","Data":"e3933aa98b22e82394e3bf58a9e85f56e501c45bbfaf1f467074dd646be1bebb"} Jul 08 18:18:43.164301 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:43.164268 2565 scope.go:117] "RemoveContainer" containerID="ff8f82708b05b50d9821dbcd04172e7d594f5cab3ae44d8a659a1d81af1cd8f6" Jul 08 18:18:43.164581 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:43.164563 2565 scope.go:117] "RemoveContainer" containerID="e3933aa98b22e82394e3bf58a9e85f56e501c45bbfaf1f467074dd646be1bebb" Jul 08 18:18:43.164788 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:43.164760 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:18:44.069246 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:44.069218 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ggfnv" Jul 08 18:18:44.168279 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:44.168259 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/2.log" Jul 08 18:18:50.028307 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:50.028276 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:50.028694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:50.028324 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:18:50.028694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:50.028587 2565 scope.go:117] "RemoveContainer" containerID="e3933aa98b22e82394e3bf58a9e85f56e501c45bbfaf1f467074dd646be1bebb" Jul 08 18:18:50.028764 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:50.028741 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:18:51.505674 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.505631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:18:51.506110 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.505736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:51.508915 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.508896 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 18:18:51.508961 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.508896 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 18:18:51.516479 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.516459 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 18:18:51.516533 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.516522 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:19:55.516499916 +0000 UTC m=+130.140086987 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : secret "metrics-daemon-secret" not found Jul 08 18:18:51.518510 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.518491 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38d4dce-6648-426f-98f7-d2fbaf4e89df-original-pull-secret\") pod \"global-pull-secret-syncer-v7kp9\" (UID: \"d38d4dce-6648-426f-98f7-d2fbaf4e89df\") " pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:51.606094 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.606063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:18:51.606094 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.606098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:51.606246 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.606126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:18:51.606246 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.606156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:18:51.606246 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.606194 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:18:51.606246 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.606211 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:18:51.606246 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.606222 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:18:51.606391 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.606254 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:19:23.606241723 +0000 UTC m=+98.229828781 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:18:51.606391 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.606256 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:18:51.606391 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.606276 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:19:23.606259854 +0000 UTC m=+98.229846937 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:18:51.606391 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:18:51.606293 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:19:23.606284699 +0000 UTC m=+98.229871786 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:18:51.608420 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.608398 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nhs\" (UniqueName: \"kubernetes.io/projected/a61af694-9ea9-4f7b-8509-cb463bb5d73b-kube-api-access-n9nhs\") pod \"network-check-target-pfzrd\" (UID: \"a61af694-9ea9-4f7b-8509-cb463bb5d73b\") " pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:51.733371 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.733344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7kp9" Jul 08 18:18:51.743556 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.743528 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kcpkw\"" Jul 08 18:18:51.751864 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.751846 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:51.861382 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.861354 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7kp9"] Jul 08 18:18:51.864102 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:18:51.864078 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38d4dce_6648_426f_98f7_d2fbaf4e89df.slice/crio-8a14947494e8df08164c0e67c237933d80632b0a099d211555048cc7e2aac744 WatchSource:0}: Error finding container 8a14947494e8df08164c0e67c237933d80632b0a099d211555048cc7e2aac744: Status 404 returned error can't find the container with id 8a14947494e8df08164c0e67c237933d80632b0a099d211555048cc7e2aac744 Jul 08 18:18:51.874917 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:51.874889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pfzrd"] Jul 08 18:18:51.877357 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:18:51.877326 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61af694_9ea9_4f7b_8509_cb463bb5d73b.slice/crio-2a201d1f17cdcd8994f02663e0770ef42676fbd22371e160ffc67ab34e3f9651 WatchSource:0}: Error finding container 2a201d1f17cdcd8994f02663e0770ef42676fbd22371e160ffc67ab34e3f9651: Status 404 returned error can't find the container with id 2a201d1f17cdcd8994f02663e0770ef42676fbd22371e160ffc67ab34e3f9651 Jul 08 18:18:52.190328 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:52.190253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pfzrd" event={"ID":"a61af694-9ea9-4f7b-8509-cb463bb5d73b","Type":"ContainerStarted","Data":"ecb3a6d2831b3096daa4f187bd14974ac54e885453e46a0d3f5f94e80e632f76"} Jul 08 18:18:52.190328 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:52.190310 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pfzrd" event={"ID":"a61af694-9ea9-4f7b-8509-cb463bb5d73b","Type":"ContainerStarted","Data":"2a201d1f17cdcd8994f02663e0770ef42676fbd22371e160ffc67ab34e3f9651"} Jul 08 18:18:52.190543 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:52.190370 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:18:52.191311 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:52.191287 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7kp9" event={"ID":"d38d4dce-6648-426f-98f7-d2fbaf4e89df","Type":"ContainerStarted","Data":"8a14947494e8df08164c0e67c237933d80632b0a099d211555048cc7e2aac744"} Jul 08 18:18:52.212944 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:52.210840 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pfzrd" podStartSLOduration=66.210823443 podStartE2EDuration="1m6.210823443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:52.208907878 +0000 UTC m=+66.832494958" watchObservedRunningTime="2026-07-08 18:18:52.210823443 +0000 UTC m=+66.834410524" Jul 08 18:18:57.210058 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:57.210024 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7kp9" event={"ID":"d38d4dce-6648-426f-98f7-d2fbaf4e89df","Type":"ContainerStarted","Data":"f267bcdd3ca61d99276caae7bd321d631924825b718d55abb33b8f7df1eded57"} Jul 08 18:18:57.226043 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:18:57.226000 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v7kp9" podStartSLOduration=66.386512518 podStartE2EDuration="1m11.225987791s" totalImagesPullingTime="4.839475273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:57.225443776 +0000 UTC m=+71.849030866" watchObservedRunningTime="2026-07-08 18:18:57.225987791 +0000 UTC m=+71.849574933" Jul 08 18:19:01.921709 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:01.921681 2565 scope.go:117] "RemoveContainer" containerID="e3933aa98b22e82394e3bf58a9e85f56e501c45bbfaf1f467074dd646be1bebb" Jul 08 18:19:01.922146 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:01.921821 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:19:13.922233 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:13.922125 2565 scope.go:117] "RemoveContainer" containerID="e3933aa98b22e82394e3bf58a9e85f56e501c45bbfaf1f467074dd646be1bebb" Jul 08 18:19:14.263533 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:14.263508 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/3.log" Jul 08 18:19:14.263932 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:14.263914 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/2.log" Jul 08 18:19:14.264052 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:14.263951 2565 generic.go:356] "Generic (PLEG): container finished" podID="fac2c03c-dff8-4df8-8400-0e30266cabfa" containerID="7eeb421d4d02a86b56062e7f48750ad7b1467c8af5a72fc48b3e8672c3e85581" exitCode=255 Jul 08 18:19:14.264052 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:14.263988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerDied","Data":"7eeb421d4d02a86b56062e7f48750ad7b1467c8af5a72fc48b3e8672c3e85581"} Jul 08 18:19:14.264052 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:14.264045 2565 scope.go:117] "RemoveContainer" containerID="e3933aa98b22e82394e3bf58a9e85f56e501c45bbfaf1f467074dd646be1bebb" Jul 08 18:19:14.264334 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:14.264309 2565 scope.go:117] "RemoveContainer" containerID="7eeb421d4d02a86b56062e7f48750ad7b1467c8af5a72fc48b3e8672c3e85581" Jul 08 18:19:14.264494 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:14.264477 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:19:15.267968 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:15.267941 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/3.log" Jul 08 18:19:20.027896 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:20.027864 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:19:20.028268 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:20.027919 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:19:20.028268 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:20.028172 2565 scope.go:117] "RemoveContainer" containerID="7eeb421d4d02a86b56062e7f48750ad7b1467c8af5a72fc48b3e8672c3e85581" Jul 08 18:19:20.028337 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:20.028296 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:19:23.197674 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:23.197640 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pfzrd" Jul 08 18:19:23.627265 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:23.627231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:19:23.627392 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:23.627286 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:19:23.627392 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:23.627317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:19:23.627392 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:23.627373 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:19:23.627493 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:23.627394 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:19:23.627493 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:23.627430 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:20:27.627415751 +0000 UTC m=+162.251002813 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:19:23.627493 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:23.627443 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:20:27.627437712 +0000 UTC m=+162.251024771 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:19:23.627493 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:23.627444 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:19:23.627493 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:23.627462 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:19:23.627646 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:23.627515 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:20:27.627501347 +0000 UTC m=+162.251088422 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:19:34.921611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:34.921576 2565 scope.go:117] "RemoveContainer" containerID="7eeb421d4d02a86b56062e7f48750ad7b1467c8af5a72fc48b3e8672c3e85581" Jul 08 18:19:34.922183 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:34.921795 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:19:47.922260 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:47.922231 2565 scope.go:117] "RemoveContainer" containerID="7eeb421d4d02a86b56062e7f48750ad7b1467c8af5a72fc48b3e8672c3e85581" Jul 08 18:19:47.922752 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:47.922378 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:19:55.546679 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:55.546620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:19:55.547154 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:55.546770 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 18:19:55.547154 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:55.546843 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:21:57.546827551 +0000 UTC m=+252.170414609 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : secret "metrics-daemon-secret" not found Jul 08 18:19:58.922372 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:58.922346 2565 scope.go:117] "RemoveContainer" containerID="7eeb421d4d02a86b56062e7f48750ad7b1467c8af5a72fc48b3e8672c3e85581" Jul 08 18:19:59.393961 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:59.393938 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/4.log" Jul 08 18:19:59.394322 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:59.394309 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/3.log" Jul 08 18:19:59.394368 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:59.394341 2565 generic.go:356] "Generic (PLEG): container finished" podID="fac2c03c-dff8-4df8-8400-0e30266cabfa" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" exitCode=255 Jul 08 18:19:59.394410 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:59.394397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerDied","Data":"d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df"} Jul 08 18:19:59.394475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:59.394465 2565 scope.go:117] "RemoveContainer" containerID="7eeb421d4d02a86b56062e7f48750ad7b1467c8af5a72fc48b3e8672c3e85581" Jul 08 18:19:59.394760 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:19:59.394740 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:19:59.394934 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:19:59.394914 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:20:00.027973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:00.027937 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:20:00.028330 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:00.027992 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:20:00.398653 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:00.398584 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/4.log" Jul 08 18:20:00.398912 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:00.398899 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:20:00.399066 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:00.399049 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:20:14.921981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:14.921950 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:20:14.922480 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:14.922113 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:20:22.702810 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:22.702767 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" podUID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" Jul 08 18:20:22.712906 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:22.712880 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" podUID="a6e10b2d-d75a-41ba-993e-35bfe3217286" Jul 08 18:20:22.758343 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:22.758319 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dn82h" podUID="73787897-ffc8-47cb-befc-1d8af360d229" Jul 08 18:20:22.947469 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:22.947434 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:20:23.460800 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:23.460775 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dn82h" Jul 08 18:20:23.460928 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:23.460775 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:20:23.460973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:23.460784 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:20:26.922479 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:26.922448 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:20:26.922881 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:26.922612 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:20:27.668539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:27.668508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:20:27.668747 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:27.668550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:20:27.668747 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:27.668583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:20:27.668747 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:27.668673 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:20:27.668747 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:27.668683 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:20:27.668747 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:27.668699 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c454f47b9-658l2: secret "image-registry-tls" not found Jul 08 18:20:27.668747 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:27.668682 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:20:27.668747 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:27.668738 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls podName:09e31dfd-bf6f-446a-af68-bc4c01a1e861 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:29.668725183 +0000 UTC m=+284.292312254 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls") pod "image-registry-6c454f47b9-658l2" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861") : secret "image-registry-tls" not found Jul 08 18:20:27.668989 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:27.668758 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:29.668743018 +0000 UTC m=+284.292330080 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:20:27.668989 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:27.668772 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls podName:73787897-ffc8-47cb-befc-1d8af360d229 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:29.668765297 +0000 UTC m=+284.292352355 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls") pod "dns-default-dn82h" (UID: "73787897-ffc8-47cb-befc-1d8af360d229") : secret "dns-default-metrics-tls" not found Jul 08 18:20:34.921638 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:34.921596 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:20:41.921635 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:41.921558 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:20:41.922082 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:41.921747 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:20:52.922160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:20:52.922125 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:20:52.922707 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:20:52.922296 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:21:03.922361 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:03.922319 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:21:03.923048 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:21:03.922481 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:21:18.922233 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:18.922200 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:21:18.922770 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:21:18.922370 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:21:29.922625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:29.922593 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:21:30.660859 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:30.660833 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:21:30.661244 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:30.661229 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/4.log" Jul 08 18:21:30.661292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:30.661264 2565 generic.go:356] "Generic (PLEG): container finished" podID="fac2c03c-dff8-4df8-8400-0e30266cabfa" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" exitCode=255 Jul 08 18:21:30.661331 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:30.661319 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerDied","Data":"b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610"} Jul 08 18:21:30.661372 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:30.661363 2565 scope.go:117] "RemoveContainer" containerID="d94e76d7414a8fc763144f5685bc7712c6afff11c0f65951fd109726e433e7df" Jul 08 18:21:30.661646 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:30.661629 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:21:30.661840 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:21:30.661820 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:21:31.665715 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:31.665690 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:21:40.028687 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:40.028648 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:21:40.029187 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:40.028704 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:21:40.029187 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:40.028937 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:21:40.029187 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:21:40.029057 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:21:50.921715 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:50.921685 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:21:50.922181 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:21:50.921845 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:21:57.565774 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:21:57.565741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:21:57.566223 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:21:57.565857 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 18:21:57.566223 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:21:57.565918 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs podName:79fe673d-c3eb-4eda-8558-6498fbd55bd7 nodeName:}" failed. No retries permitted until 2026-07-08 18:23:59.565904568 +0000 UTC m=+374.189491626 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs") pod "network-metrics-daemon-4dl6h" (UID: "79fe673d-c3eb-4eda-8558-6498fbd55bd7") : secret "metrics-daemon-secret" not found Jul 08 18:22:04.922124 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:04.922099 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:22:04.922582 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:04.922238 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:22:13.650905 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.650688 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742"] Jul 08 18:22:13.654971 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.654948 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn"] Jul 08 18:22:13.655114 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.655094 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:13.658422 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.658400 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.659735 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.659718 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:22:13.659857 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.659735 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 18:22:13.659857 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.659736 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-884fq\"" Jul 08 18:22:13.659857 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.659722 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 18:22:13.660325 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.660311 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 18:22:13.660878 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.660861 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-24nn7\"" Jul 08 18:22:13.660977 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.660962 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 18:22:13.661186 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.661172 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 18:22:13.662320 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.662102 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:22:13.675452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.675427 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn"] Jul 08 18:22:13.676240 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.676221 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742"] Jul 08 18:22:13.767008 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.766986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38444fe4-2b88-467a-bd50-8183c175e6ed-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.767100 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.767017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k24rn\" (UniqueName: \"kubernetes.io/projected/1c878d8f-3614-4fa6-85ab-55b0574d1c26-kube-api-access-k24rn\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:13.767100 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.767037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38444fe4-2b88-467a-bd50-8183c175e6ed-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.767176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.767135 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:13.767176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.767168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5gds\" (UniqueName: \"kubernetes.io/projected/38444fe4-2b88-467a-bd50-8183c175e6ed-kube-api-access-j5gds\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.868127 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.868101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:13.868234 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.868139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5gds\" (UniqueName: \"kubernetes.io/projected/38444fe4-2b88-467a-bd50-8183c175e6ed-kube-api-access-j5gds\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.868234 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.868176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38444fe4-2b88-467a-bd50-8183c175e6ed-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.868234 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.868199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k24rn\" (UniqueName: \"kubernetes.io/projected/1c878d8f-3614-4fa6-85ab-55b0574d1c26-kube-api-access-k24rn\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:13.868234 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.868216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38444fe4-2b88-467a-bd50-8183c175e6ed-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.868411 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:13.868394 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:22:13.868490 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:13.868477 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls podName:1c878d8f-3614-4fa6-85ab-55b0574d1c26 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:14.368457277 +0000 UTC m=+268.992044343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29742" (UID: "1c878d8f-3614-4fa6-85ab-55b0574d1c26") : secret "samples-operator-tls" not found Jul 08 18:22:13.868821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.868800 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38444fe4-2b88-467a-bd50-8183c175e6ed-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.870447 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.870424 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38444fe4-2b88-467a-bd50-8183c175e6ed-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.878919 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.878900 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5gds\" (UniqueName: \"kubernetes.io/projected/38444fe4-2b88-467a-bd50-8183c175e6ed-kube-api-access-j5gds\") pod \"kube-storage-version-migrator-operator-846bf668f7-4dpmn\" (UID: \"38444fe4-2b88-467a-bd50-8183c175e6ed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:13.879382 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.879363 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k24rn\" (UniqueName: \"kubernetes.io/projected/1c878d8f-3614-4fa6-85ab-55b0574d1c26-kube-api-access-k24rn\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:13.970222 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:13.970173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" Jul 08 18:22:14.086056 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:14.086027 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn"] Jul 08 18:22:14.088459 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:14.088436 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38444fe4_2b88_467a_bd50_8183c175e6ed.slice/crio-5c8103356865102dfc0aef2f2000f8c526b471b7a05da93c8acfa22e90f86d5b WatchSource:0}: Error finding container 5c8103356865102dfc0aef2f2000f8c526b471b7a05da93c8acfa22e90f86d5b: Status 404 returned error can't find the container with id 5c8103356865102dfc0aef2f2000f8c526b471b7a05da93c8acfa22e90f86d5b Jul 08 18:22:14.372428 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:14.372402 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:14.372562 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:14.372545 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:22:14.372612 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:14.372604 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls podName:1c878d8f-3614-4fa6-85ab-55b0574d1c26 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:15.372589239 +0000 UTC m=+269.996176300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29742" (UID: "1c878d8f-3614-4fa6-85ab-55b0574d1c26") : secret "samples-operator-tls" not found Jul 08 18:22:14.793679 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:14.793629 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" event={"ID":"38444fe4-2b88-467a-bd50-8183c175e6ed","Type":"ContainerStarted","Data":"5c8103356865102dfc0aef2f2000f8c526b471b7a05da93c8acfa22e90f86d5b"} Jul 08 18:22:15.379881 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:15.379848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:15.380042 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:15.379994 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:22:15.380083 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:15.380055 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls podName:1c878d8f-3614-4fa6-85ab-55b0574d1c26 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:17.380038633 +0000 UTC m=+272.003625707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29742" (UID: "1c878d8f-3614-4fa6-85ab-55b0574d1c26") : secret "samples-operator-tls" not found Jul 08 18:22:16.801072 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:16.801038 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" event={"ID":"38444fe4-2b88-467a-bd50-8183c175e6ed","Type":"ContainerStarted","Data":"3a25ce25c1678104b60322eb97f4cedb12373d534e190571d443a9b1eb12133a"} Jul 08 18:22:16.821450 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:16.821403 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" podStartSLOduration=1.911549928 podStartE2EDuration="3.82138704s" totalImagesPullingTime="1.909837112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:16.820947706 +0000 UTC m=+271.444534821" watchObservedRunningTime="2026-07-08 18:22:16.82138704 +0000 UTC m=+271.444974125" Jul 08 18:22:16.921999 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:16.921970 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:22:16.922138 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:16.922097 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:22:17.013243 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.013209 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ttw49"] Jul 08 18:22:17.016506 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.016484 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.019728 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.019709 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 18:22:17.019826 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.019805 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 18:22:17.019884 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.019825 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7xvt\"" Jul 08 18:22:17.020036 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.020022 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 18:22:17.020247 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.020235 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 18:22:17.032066 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.032047 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ttw49"] Jul 08 18:22:17.091838 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.091787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4cac3c2-5442-4580-9d4c-9bf130fa8382-crio-socket\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.091838 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.091831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdjh\" (UniqueName: \"kubernetes.io/projected/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-api-access-kpdjh\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.091974 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.091870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.091974 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.091910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.092045 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.092018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4cac3c2-5442-4580-9d4c-9bf130fa8382-data-volume\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.192911 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.192887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4cac3c2-5442-4580-9d4c-9bf130fa8382-data-volume\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.192996 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.192926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4cac3c2-5442-4580-9d4c-9bf130fa8382-crio-socket\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.192996 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.192962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdjh\" (UniqueName: \"kubernetes.io/projected/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-api-access-kpdjh\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.193117 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.193000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.193117 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.193038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.193117 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.193078 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4cac3c2-5442-4580-9d4c-9bf130fa8382-crio-socket\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.193261 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.193142 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:22:17.193261 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.193150 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:22:17.193261 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.193220 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:17.693200885 +0000 UTC m=+272.316787955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : configmap "kube-rbac-proxy" not found Jul 08 18:22:17.193261 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.193222 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4cac3c2-5442-4580-9d4c-9bf130fa8382-data-volume\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.193261 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.193241 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:17.693230441 +0000 UTC m=+272.316817513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : secret "insights-runtime-extractor-tls" not found Jul 08 18:22:17.203051 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.203028 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdjh\" (UniqueName: \"kubernetes.io/projected/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-api-access-kpdjh\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.394353 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.394297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:17.394458 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.394396 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:22:17.394458 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.394457 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls podName:1c878d8f-3614-4fa6-85ab-55b0574d1c26 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:21.394443855 +0000 UTC m=+276.018030913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29742" (UID: "1c878d8f-3614-4fa6-85ab-55b0574d1c26") : secret "samples-operator-tls" not found Jul 08 18:22:17.569647 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.569620 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln"] Jul 08 18:22:17.572648 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.572632 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" Jul 08 18:22:17.576007 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.575983 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 08 18:22:17.576121 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.576071 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 08 18:22:17.576121 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.576103 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mg6v8\"" Jul 08 18:22:17.582564 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.582545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln"] Jul 08 18:22:17.596055 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.596033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9wc6\" (UniqueName: \"kubernetes.io/projected/7230e111-6e2c-4076-92b2-5121dd1999e1-kube-api-access-x9wc6\") pod \"migrator-d57d58cc9-xvxln\" (UID: \"7230e111-6e2c-4076-92b2-5121dd1999e1\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" Jul 08 18:22:17.696569 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.696507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9wc6\" (UniqueName: \"kubernetes.io/projected/7230e111-6e2c-4076-92b2-5121dd1999e1-kube-api-access-x9wc6\") pod \"migrator-d57d58cc9-xvxln\" (UID: \"7230e111-6e2c-4076-92b2-5121dd1999e1\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" Jul 08 18:22:17.696697 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.696578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.696697 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.696620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:17.696813 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.696738 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:22:17.696813 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.696754 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:22:17.696813 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.696793 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:18.696779503 +0000 UTC m=+273.320366561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : secret "insights-runtime-extractor-tls" not found Jul 08 18:22:17.696930 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:17.696825 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:18.696808181 +0000 UTC m=+273.320395254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : configmap "kube-rbac-proxy" not found Jul 08 18:22:17.704211 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.704185 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9wc6\" (UniqueName: \"kubernetes.io/projected/7230e111-6e2c-4076-92b2-5121dd1999e1-kube-api-access-x9wc6\") pod \"migrator-d57d58cc9-xvxln\" (UID: \"7230e111-6e2c-4076-92b2-5121dd1999e1\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" Jul 08 18:22:17.881355 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:17.881336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" Jul 08 18:22:18.002723 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:18.002630 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln"] Jul 08 18:22:18.004547 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:18.004525 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7230e111_6e2c_4076_92b2_5121dd1999e1.slice/crio-229116b3ad03cf6e0ad61fc9150fd4ea44fd99c1040998156909d031d77788c1 WatchSource:0}: Error finding container 229116b3ad03cf6e0ad61fc9150fd4ea44fd99c1040998156909d031d77788c1: Status 404 returned error can't find the container with id 229116b3ad03cf6e0ad61fc9150fd4ea44fd99c1040998156909d031d77788c1 Jul 08 18:22:18.703926 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:18.703896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:18.704103 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:18.703943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:18.704103 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:18.704016 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:22:18.704103 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:18.704037 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:22:18.704103 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:18.704076 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:20.704062505 +0000 UTC m=+275.327649564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : configmap "kube-rbac-proxy" not found Jul 08 18:22:18.704103 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:18.704099 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:20.704084758 +0000 UTC m=+275.327671816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : secret "insights-runtime-extractor-tls" not found Jul 08 18:22:18.808163 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:18.808131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" event={"ID":"7230e111-6e2c-4076-92b2-5121dd1999e1","Type":"ContainerStarted","Data":"229116b3ad03cf6e0ad61fc9150fd4ea44fd99c1040998156909d031d77788c1"} Jul 08 18:22:19.079603 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:19.079583 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c228s_d28adaf6-6f30-4afd-80cf-191bc1b23cc0/dns-node-resolver/0.log" Jul 08 18:22:19.812956 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:19.812924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" event={"ID":"7230e111-6e2c-4076-92b2-5121dd1999e1","Type":"ContainerStarted","Data":"d43ddb3f560d1024768e84032f2d0b9262b4b081e03623dbf77a1a04838dda05"} Jul 08 18:22:19.813158 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:19.812978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" event={"ID":"7230e111-6e2c-4076-92b2-5121dd1999e1","Type":"ContainerStarted","Data":"8b6d4fe39ac90e2e84fa773823e1fffba06f4194992e76042f71c610d56a7878"} Jul 08 18:22:19.831594 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:19.831547 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xvxln" podStartSLOduration=1.88082926 podStartE2EDuration="2.831530206s" totalImagesPullingTime="950.700946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:19.829897532 +0000 UTC m=+274.453484625" watchObservedRunningTime="2026-07-08 18:22:19.831530206 +0000 UTC m=+274.455117285" Jul 08 18:22:19.882598 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:19.882570 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjk7q_9e769af3-83dc-4841-95a4-b514d5e6b179/node-ca/0.log" Jul 08 18:22:20.523656 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.523628 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-99qrw"] Jul 08 18:22:20.531143 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.531124 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.533811 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.533794 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 18:22:20.534127 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.534107 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 18:22:20.535079 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.535017 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 18:22:20.535182 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.535088 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 18:22:20.535182 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.535109 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vdmkr\"" Jul 08 18:22:20.536442 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.536422 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-99qrw"] Jul 08 18:22:20.617419 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.617393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-signing-key\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.617516 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.617425 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-signing-cabundle\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.617566 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.617511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzk6w\" (UniqueName: \"kubernetes.io/projected/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-kube-api-access-tzk6w\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.680555 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.680534 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5d65v_e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4/serve-healthcheck-canary/0.log" Jul 08 18:22:20.717990 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.717969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:20.718046 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.717994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-signing-key\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.718046 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.718013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-signing-cabundle\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.718046 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.718034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:20.718139 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:20.718110 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:22:20.718139 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.718133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzk6w\" (UniqueName: \"kubernetes.io/projected/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-kube-api-access-tzk6w\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.718222 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:20.718172 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:24.718154743 +0000 UTC m=+279.341741855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : secret "insights-runtime-extractor-tls" not found Jul 08 18:22:20.718222 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:20.718183 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:22:20.718304 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:20.718246 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:24.718232224 +0000 UTC m=+279.341819296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : configmap "kube-rbac-proxy" not found Jul 08 18:22:20.718777 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.718751 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-signing-cabundle\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.720551 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.720529 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-signing-key\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.726710 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.726686 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzk6w\" (UniqueName: \"kubernetes.io/projected/717f92aa-d9a8-4685-82ea-9a5ac9f1ced7-kube-api-access-tzk6w\") pod \"service-ca-758f599c5-99qrw\" (UID: \"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7\") " pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.841435 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.841387 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-99qrw" Jul 08 18:22:20.964071 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:20.964017 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-99qrw"] Jul 08 18:22:20.967923 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:20.967892 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717f92aa_d9a8_4685_82ea_9a5ac9f1ced7.slice/crio-dc1f6b075edffd6ec1566b3d1331dd8c4a5a49e91675bdcc4b49efc9fadab83d WatchSource:0}: Error finding container dc1f6b075edffd6ec1566b3d1331dd8c4a5a49e91675bdcc4b49efc9fadab83d: Status 404 returned error can't find the container with id dc1f6b075edffd6ec1566b3d1331dd8c4a5a49e91675bdcc4b49efc9fadab83d Jul 08 18:22:21.422869 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:21.422838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:21.423017 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:21.422988 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:22:21.423074 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:21.423054 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls podName:1c878d8f-3614-4fa6-85ab-55b0574d1c26 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:29.423038673 +0000 UTC m=+284.046625731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29742" (UID: "1c878d8f-3614-4fa6-85ab-55b0574d1c26") : secret "samples-operator-tls" not found Jul 08 18:22:21.821307 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:21.821277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-99qrw" event={"ID":"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7","Type":"ContainerStarted","Data":"dc1f6b075edffd6ec1566b3d1331dd8c4a5a49e91675bdcc4b49efc9fadab83d"} Jul 08 18:22:23.829695 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:23.829645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-99qrw" event={"ID":"717f92aa-d9a8-4685-82ea-9a5ac9f1ced7","Type":"ContainerStarted","Data":"b16355d045b1f0ef1c560206aa98055b125ac2283450695b4a7992224b36cf40"} Jul 08 18:22:23.852625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:23.852575 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-99qrw" podStartSLOduration=1.631335321 podStartE2EDuration="3.852554378s" totalImagesPullingTime="2.221219057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:23.851732301 +0000 UTC m=+278.475319382" watchObservedRunningTime="2026-07-08 18:22:23.852554378 +0000 UTC m=+278.476141463" Jul 08 18:22:24.750972 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:24.750935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:24.751157 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:24.750990 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:24.751157 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:24.751104 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:22:24.751261 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:24.751176 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:32.751156612 +0000 UTC m=+287.374743672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : secret "insights-runtime-extractor-tls" not found Jul 08 18:22:24.751357 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:24.751113 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:22:24.751491 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:24.751474 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:32.751454435 +0000 UTC m=+287.375041507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : configmap "kube-rbac-proxy" not found Jul 08 18:22:26.461696 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:26.461646 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" podUID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" Jul 08 18:22:26.461696 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:26.461689 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dn82h" podUID="73787897-ffc8-47cb-befc-1d8af360d229" Jul 08 18:22:26.462080 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:26.461692 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" podUID="a6e10b2d-d75a-41ba-993e-35bfe3217286" Jul 08 18:22:26.840925 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:26.840906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:22:26.841079 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:26.840910 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:22:26.841079 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:26.840914 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dn82h" Jul 08 18:22:29.488149 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.488123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:29.490436 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.490406 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c878d8f-3614-4fa6-85ab-55b0574d1c26-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29742\" (UID: \"1c878d8f-3614-4fa6-85ab-55b0574d1c26\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:29.564975 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.564944 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" Jul 08 18:22:29.687354 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.687332 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742"] Jul 08 18:22:29.690368 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.690347 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:22:29.690445 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.690392 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:22:29.690512 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:29.690499 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:22:29.690572 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:29.690562 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert podName:a6e10b2d-d75a-41ba-993e-35bfe3217286 nodeName:}" failed. No retries permitted until 2026-07-08 18:24:31.69054422 +0000 UTC m=+406.314131295 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-xlmdv" (UID: "a6e10b2d-d75a-41ba-993e-35bfe3217286") : secret "networking-console-plugin-cert" not found Jul 08 18:22:29.690616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.690587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:22:29.692508 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.692488 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/73787897-ffc8-47cb-befc-1d8af360d229-metrics-tls\") pod \"dns-default-dn82h\" (UID: \"73787897-ffc8-47cb-befc-1d8af360d229\") " pod="openshift-dns/dns-default-dn82h" Jul 08 18:22:29.692752 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.692734 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"image-registry-6c454f47b9-658l2\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:22:29.844045 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.843990 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hm9k9\"" Jul 08 18:22:29.844045 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.844026 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4sdr\"" Jul 08 18:22:29.849654 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.849635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" event={"ID":"1c878d8f-3614-4fa6-85ab-55b0574d1c26","Type":"ContainerStarted","Data":"29861ba761c92eb19124128ef7aba2b8d1a667ea06965097d9f88b7499349cc0"} Jul 08 18:22:29.851901 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.851882 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:22:29.851955 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.851917 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dn82h" Jul 08 18:22:29.986531 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:29.986506 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dn82h"] Jul 08 18:22:29.989760 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:29.989732 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73787897_ffc8_47cb_befc_1d8af360d229.slice/crio-d071914938992ca824806ed794b2deea1712aa359d2e4d3132873c505dd42787 WatchSource:0}: Error finding container d071914938992ca824806ed794b2deea1712aa359d2e4d3132873c505dd42787: Status 404 returned error can't find the container with id d071914938992ca824806ed794b2deea1712aa359d2e4d3132873c505dd42787 Jul 08 18:22:30.003251 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:30.003230 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c454f47b9-658l2"] Jul 08 18:22:30.004793 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:30.004768 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e31dfd_bf6f_446a_af68_bc4c01a1e861.slice/crio-95619ee8900dc720e1303e6e2501d666405847e831df8ea1bf63dc7caba2825d WatchSource:0}: Error finding container 95619ee8900dc720e1303e6e2501d666405847e831df8ea1bf63dc7caba2825d: Status 404 returned error can't find the container with id 95619ee8900dc720e1303e6e2501d666405847e831df8ea1bf63dc7caba2825d Jul 08 18:22:30.861021 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:30.860980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" event={"ID":"09e31dfd-bf6f-446a-af68-bc4c01a1e861","Type":"ContainerStarted","Data":"5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8"} Jul 08 18:22:30.861482 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:30.861041 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" event={"ID":"09e31dfd-bf6f-446a-af68-bc4c01a1e861","Type":"ContainerStarted","Data":"95619ee8900dc720e1303e6e2501d666405847e831df8ea1bf63dc7caba2825d"} Jul 08 18:22:30.861482 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:30.861092 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:22:30.863190 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:30.863057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dn82h" event={"ID":"73787897-ffc8-47cb-befc-1d8af360d229","Type":"ContainerStarted","Data":"d071914938992ca824806ed794b2deea1712aa359d2e4d3132873c505dd42787"} Jul 08 18:22:30.882819 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:30.882757 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" podStartSLOduration=284.882737001 podStartE2EDuration="4m44.882737001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:30.88141283 +0000 UTC m=+285.504999909" watchObservedRunningTime="2026-07-08 18:22:30.882737001 +0000 UTC m=+285.506324081" Jul 08 18:22:30.922054 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:30.922030 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:22:30.922245 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:30.922221 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:22:32.819113 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:32.819075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:32.819567 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:32.819143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:32.819567 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:32.819222 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:22:32.819567 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:32.819294 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls podName:d4cac3c2-5442-4580-9d4c-9bf130fa8382 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:48.819272157 +0000 UTC m=+303.442859217 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttw49" (UID: "d4cac3c2-5442-4580-9d4c-9bf130fa8382") : secret "insights-runtime-extractor-tls" not found Jul 08 18:22:32.819813 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:32.819792 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4cac3c2-5442-4580-9d4c-9bf130fa8382-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:33.874230 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:33.874198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dn82h" event={"ID":"73787897-ffc8-47cb-befc-1d8af360d229","Type":"ContainerStarted","Data":"308eafce9e7039db9ccccdae97e47e9a7ab5e6803f4b7d70985f4532c82eeee6"} Jul 08 18:22:33.874678 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:33.874247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dn82h" event={"ID":"73787897-ffc8-47cb-befc-1d8af360d229","Type":"ContainerStarted","Data":"8b3343c70979687f2fec986b3554e9aaf378fbe5f211b96e8411e175f2592b94"} Jul 08 18:22:33.874678 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:33.874290 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dn82h" Jul 08 18:22:33.876004 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:33.875980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" event={"ID":"1c878d8f-3614-4fa6-85ab-55b0574d1c26","Type":"ContainerStarted","Data":"0e90c6ca325ef2d8a792f92e1f8ff1a917dacacd963f68e7827b5f7b34e21beb"} Jul 08 18:22:33.876117 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:33.876024 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" event={"ID":"1c878d8f-3614-4fa6-85ab-55b0574d1c26","Type":"ContainerStarted","Data":"198608aca5f70ca938721334e261ecbfe8b7e934df6454631bfe9cf2f3c9ee28"} Jul 08 18:22:33.910168 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:33.910120 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dn82h" podStartSLOduration=251.764438204 podStartE2EDuration="4m14.91010914s" totalImagesPullingTime="3.145670936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:33.90899306 +0000 UTC m=+288.532580160" watchObservedRunningTime="2026-07-08 18:22:33.91010914 +0000 UTC m=+288.533696220" Jul 08 18:22:33.928317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:33.928278 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29742" podStartSLOduration=17.523316622 podStartE2EDuration="20.928267159s" totalImagesPullingTime="3.404950537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:33.927737137 +0000 UTC m=+288.551324218" watchObservedRunningTime="2026-07-08 18:22:33.928267159 +0000 UTC m=+288.551854238" Jul 08 18:22:37.923021 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:37.922980 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4dl6h" podUID="79fe673d-c3eb-4eda-8558-6498fbd55bd7" Jul 08 18:22:43.882031 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:43.881996 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dn82h" Jul 08 18:22:43.922429 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:43.922406 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:22:43.922583 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:43.922567 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:22:45.712558 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:45.712518 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:22:45.713137 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:45.712822 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:22:45.715225 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:45.715203 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:22:45.715442 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:45.715420 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:22:45.723153 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:45.723132 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 18:22:48.570314 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.570279 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8"] Jul 08 18:22:48.573528 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.573505 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" Jul 08 18:22:48.575887 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.575866 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 18:22:48.575990 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.575886 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gxwnn\"" Jul 08 18:22:48.584566 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.584543 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8"] Jul 08 18:22:48.629364 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.629335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e68efccd-724b-487e-89ac-e3dc8244e5f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-tb8v8\" (UID: \"e68efccd-724b-487e-89ac-e3dc8244e5f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" Jul 08 18:22:48.729936 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.729911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e68efccd-724b-487e-89ac-e3dc8244e5f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-tb8v8\" (UID: \"e68efccd-724b-487e-89ac-e3dc8244e5f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" Jul 08 18:22:48.730051 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:48.730035 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 08 18:22:48.730121 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:48.730111 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68efccd-724b-487e-89ac-e3dc8244e5f5-tls-certificates podName:e68efccd-724b-487e-89ac-e3dc8244e5f5 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:49.230097902 +0000 UTC m=+303.853684960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e68efccd-724b-487e-89ac-e3dc8244e5f5-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-tb8v8" (UID: "e68efccd-724b-487e-89ac-e3dc8244e5f5") : secret "prometheus-operator-admission-webhook-tls" not found Jul 08 18:22:48.830972 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.830915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:48.833081 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.833066 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4cac3c2-5442-4580-9d4c-9bf130fa8382-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttw49\" (UID: \"d4cac3c2-5442-4580-9d4c-9bf130fa8382\") " pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:48.921384 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:48.921354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:22:49.127791 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.127740 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7xvt\"" Jul 08 18:22:49.136006 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.135989 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ttw49" Jul 08 18:22:49.234355 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.234320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e68efccd-724b-487e-89ac-e3dc8244e5f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-tb8v8\" (UID: \"e68efccd-724b-487e-89ac-e3dc8244e5f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" Jul 08 18:22:49.236837 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.236811 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e68efccd-724b-487e-89ac-e3dc8244e5f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-tb8v8\" (UID: \"e68efccd-724b-487e-89ac-e3dc8244e5f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" Jul 08 18:22:49.256191 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.256169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ttw49"] Jul 08 18:22:49.258857 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:49.258818 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cac3c2_5442_4580_9d4c_9bf130fa8382.slice/crio-cbcedbfd34efd0eb2942b597c6e85f6c06305736dae94acab7bc47096a97d6a1 WatchSource:0}: Error finding container cbcedbfd34efd0eb2942b597c6e85f6c06305736dae94acab7bc47096a97d6a1: Status 404 returned error can't find the container with id cbcedbfd34efd0eb2942b597c6e85f6c06305736dae94acab7bc47096a97d6a1 Jul 08 18:22:49.260342 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.260326 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:22:49.482467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.482390 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" Jul 08 18:22:49.602756 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.602724 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8"] Jul 08 18:22:49.605160 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:49.605140 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68efccd_724b_487e_89ac_e3dc8244e5f5.slice/crio-fd369dcfeeba9599f0e47c7c611d5966bffb2380c41f58d1fb38b629a041dd79 WatchSource:0}: Error finding container fd369dcfeeba9599f0e47c7c611d5966bffb2380c41f58d1fb38b629a041dd79: Status 404 returned error can't find the container with id fd369dcfeeba9599f0e47c7c611d5966bffb2380c41f58d1fb38b629a041dd79 Jul 08 18:22:49.856953 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.856925 2565 patch_prober.go:28] interesting pod/image-registry-6c454f47b9-658l2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 18:22:49.857080 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.856974 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" podUID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 18:22:49.937725 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.937688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ttw49" event={"ID":"d4cac3c2-5442-4580-9d4c-9bf130fa8382","Type":"ContainerStarted","Data":"cbcedbfd34efd0eb2942b597c6e85f6c06305736dae94acab7bc47096a97d6a1"} Jul 08 18:22:49.940101 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:49.940073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" event={"ID":"e68efccd-724b-487e-89ac-e3dc8244e5f5","Type":"ContainerStarted","Data":"fd369dcfeeba9599f0e47c7c611d5966bffb2380c41f58d1fb38b629a041dd79"} Jul 08 18:22:51.872648 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:51.872482 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:22:51.947845 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:51.947814 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" event={"ID":"e68efccd-724b-487e-89ac-e3dc8244e5f5","Type":"ContainerStarted","Data":"ea091de8a2af49d4c681136be48c07c7ad3a214ab25c906c1f464c41bc0aafe1"} Jul 08 18:22:51.948012 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:51.947953 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" Jul 08 18:22:51.952943 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:51.952918 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" Jul 08 18:22:51.964372 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:51.964331 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tb8v8" podStartSLOduration=2.702107846 podStartE2EDuration="3.96431691s" totalImagesPullingTime="1.262209064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:51.96348233 +0000 UTC m=+306.587069410" watchObservedRunningTime="2026-07-08 18:22:51.96431691 +0000 UTC m=+306.587903990" Jul 08 18:22:52.640870 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.640839 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2std7"] Jul 08 18:22:52.656329 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.656303 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2std7"] Jul 08 18:22:52.656486 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.656421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.658975 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.658949 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 08 18:22:52.659219 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.659203 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 18:22:52.659534 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.659513 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 08 18:22:52.659626 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.659599 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 18:22:52.659722 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.659688 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-69khk\"" Jul 08 18:22:52.660156 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.660131 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 18:22:52.763066 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.763034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa2d397-fe25-40ae-817b-6aabb6d678a1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.763066 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.763065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faa2d397-fe25-40ae-817b-6aabb6d678a1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.763270 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.763106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faa2d397-fe25-40ae-817b-6aabb6d678a1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.763270 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.763219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgbj\" (UniqueName: \"kubernetes.io/projected/faa2d397-fe25-40ae-817b-6aabb6d678a1-kube-api-access-qjgbj\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.864096 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.864067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgbj\" (UniqueName: \"kubernetes.io/projected/faa2d397-fe25-40ae-817b-6aabb6d678a1-kube-api-access-qjgbj\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.864254 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.864154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa2d397-fe25-40ae-817b-6aabb6d678a1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.864254 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.864172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faa2d397-fe25-40ae-817b-6aabb6d678a1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.864254 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.864212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faa2d397-fe25-40ae-817b-6aabb6d678a1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.864906 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.864878 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faa2d397-fe25-40ae-817b-6aabb6d678a1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.866675 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.866640 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faa2d397-fe25-40ae-817b-6aabb6d678a1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.866853 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.866824 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa2d397-fe25-40ae-817b-6aabb6d678a1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.872210 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.872171 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgbj\" (UniqueName: \"kubernetes.io/projected/faa2d397-fe25-40ae-817b-6aabb6d678a1-kube-api-access-qjgbj\") pod \"prometheus-operator-567f7b8b7c-2std7\" (UID: \"faa2d397-fe25-40ae-817b-6aabb6d678a1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:52.968271 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:52.968217 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" Jul 08 18:22:53.624423 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:53.624225 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2std7"] Jul 08 18:22:53.626804 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:53.626779 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa2d397_fe25_40ae_817b_6aabb6d678a1.slice/crio-411d7ab6d7fe2ba0c138d5f6888ccfd5194b6ecefca287afa72e1d8829ae924c WatchSource:0}: Error finding container 411d7ab6d7fe2ba0c138d5f6888ccfd5194b6ecefca287afa72e1d8829ae924c: Status 404 returned error can't find the container with id 411d7ab6d7fe2ba0c138d5f6888ccfd5194b6ecefca287afa72e1d8829ae924c Jul 08 18:22:53.955548 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:53.955478 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ttw49" event={"ID":"d4cac3c2-5442-4580-9d4c-9bf130fa8382","Type":"ContainerStarted","Data":"711b1e1672dc5d6f93885e8719f555d8e950e7d16b9fbc1eda7b07600a3c9a3a"} Jul 08 18:22:53.956576 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:53.956555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" event={"ID":"faa2d397-fe25-40ae-817b-6aabb6d678a1","Type":"ContainerStarted","Data":"411d7ab6d7fe2ba0c138d5f6888ccfd5194b6ecefca287afa72e1d8829ae924c"} Jul 08 18:22:54.963426 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:54.963393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ttw49" event={"ID":"d4cac3c2-5442-4580-9d4c-9bf130fa8382","Type":"ContainerStarted","Data":"6e4fd220d91b316de486d159bc31b26d58ce6d5383a9072d563d0a57f8078027"} Jul 08 18:22:55.968965 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:55.968877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" event={"ID":"faa2d397-fe25-40ae-817b-6aabb6d678a1","Type":"ContainerStarted","Data":"a01b828ff1716c1a6eab6f6d45a06bfed92c246492e98de95c44cd8e86c4bd45"} Jul 08 18:22:55.968965 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:55.968943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" event={"ID":"faa2d397-fe25-40ae-817b-6aabb6d678a1","Type":"ContainerStarted","Data":"01e89667444f55710f241325b5a72a2107e860ff2714ed3e02352ccf6252ba97"} Jul 08 18:22:55.987081 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:55.987031 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2std7" podStartSLOduration=2.066543855 podStartE2EDuration="3.98701448s" totalImagesPullingTime="1.920470625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:55.984634615 +0000 UTC m=+310.608221696" watchObservedRunningTime="2026-07-08 18:22:55.98701448 +0000 UTC m=+310.610601560" Jul 08 18:22:56.921448 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:56.921420 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:22:56.921612 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:56.921592 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:22:56.974439 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:56.974414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ttw49" event={"ID":"d4cac3c2-5442-4580-9d4c-9bf130fa8382","Type":"ContainerStarted","Data":"e1a8e9338429d0e21b88b4ea36492a6396560664213344fdfb9c08f2e28a0cd8"} Jul 08 18:22:56.994624 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:56.994580 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ttw49" podStartSLOduration=33.542608283 podStartE2EDuration="40.99456669s" totalImagesPullingTime="7.451958407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:22:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:22:56.992834038 +0000 UTC m=+311.616421119" watchObservedRunningTime="2026-07-08 18:22:56.99456669 +0000 UTC m=+311.618153769" Jul 08 18:22:57.961267 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:57.961237 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf"] Jul 08 18:22:57.969118 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:57.968943 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:57.972285 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:57.972260 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 08 18:22:57.972840 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:57.972684 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6h9n8\"" Jul 08 18:22:57.972840 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:57.972723 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 08 18:22:57.976254 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:57.975999 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf"] Jul 08 18:22:58.017084 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.017055 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5dqj6"] Jul 08 18:22:58.019636 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.019621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.023504 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.023482 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 18:22:58.023791 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.023762 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 18:22:58.023913 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.023894 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 18:22:58.024175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.024159 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hbm8s\"" Jul 08 18:22:58.106794 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.106766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bafd724-094e-4ff1-9465-501558427f83-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.106909 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.106830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-root\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.106971 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.106947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8652526a-d355-4d28-bbd3-b7a70947c333-metrics-client-ca\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.107390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-wtmp\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.107498 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-sys\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.107586 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.107639 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-tls\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.107716 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.107716 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsrg\" (UniqueName: \"kubernetes.io/projected/5bafd724-094e-4ff1-9465-501558427f83-kube-api-access-hzsrg\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.107810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.107810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.107810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-textfile\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.107810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.107797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgr2k\" (UniqueName: \"kubernetes.io/projected/8652526a-d355-4d28-bbd3-b7a70947c333-kube-api-access-sgr2k\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.208545 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-sys\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.208675 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.208675 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208590 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-tls\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.208675 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.208675 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208628 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-sys\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.208882 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsrg\" (UniqueName: \"kubernetes.io/projected/5bafd724-094e-4ff1-9465-501558427f83-kube-api-access-hzsrg\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.208882 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.208882 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:58.208727 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 08 18:22:58.208882 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.208882 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-textfile\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.208882 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:58.208788 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-tls podName:5bafd724-094e-4ff1-9465-501558427f83 nodeName:}" failed. No retries permitted until 2026-07-08 18:22:58.708770691 +0000 UTC m=+313.332357754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-hnngf" (UID: "5bafd724-094e-4ff1-9465-501558427f83") : secret "openshift-state-metrics-tls" not found Jul 08 18:22:58.208882 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgr2k\" (UniqueName: \"kubernetes.io/projected/8652526a-d355-4d28-bbd3-b7a70947c333-kube-api-access-sgr2k\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.208882 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bafd724-094e-4ff1-9465-501558427f83-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.209263 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208894 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-root\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.209263 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.208961 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8652526a-d355-4d28-bbd3-b7a70947c333-metrics-client-ca\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.209263 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.209000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-wtmp\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.209263 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.209030 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-textfile\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.209263 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.209161 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-wtmp\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.209553 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.209505 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8652526a-d355-4d28-bbd3-b7a70947c333-root\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.210176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.210136 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8652526a-d355-4d28-bbd3-b7a70947c333-metrics-client-ca\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.210176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.210144 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.210317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.210252 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bafd724-094e-4ff1-9465-501558427f83-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.211570 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.211515 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-tls\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.211652 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.211624 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.211795 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.211775 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8652526a-d355-4d28-bbd3-b7a70947c333-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.221697 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.221649 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsrg\" (UniqueName: \"kubernetes.io/projected/5bafd724-094e-4ff1-9465-501558427f83-kube-api-access-hzsrg\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.221852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.221832 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgr2k\" (UniqueName: \"kubernetes.io/projected/8652526a-d355-4d28-bbd3-b7a70947c333-kube-api-access-sgr2k\") pod \"node-exporter-5dqj6\" (UID: \"8652526a-d355-4d28-bbd3-b7a70947c333\") " pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.330063 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.330029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5dqj6" Jul 08 18:22:58.341308 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:58.341246 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8652526a_d355_4d28_bbd3_b7a70947c333.slice/crio-44aeae3aac4bf48e7c401559b0d3864cbc64b3534186d167b2e5b11de058be43 WatchSource:0}: Error finding container 44aeae3aac4bf48e7c401559b0d3864cbc64b3534186d167b2e5b11de058be43: Status 404 returned error can't find the container with id 44aeae3aac4bf48e7c401559b0d3864cbc64b3534186d167b2e5b11de058be43 Jul 08 18:22:58.713730 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.713699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.716049 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.716033 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bafd724-094e-4ff1-9465-501558427f83-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-hnngf\" (UID: \"5bafd724-094e-4ff1-9465-501558427f83\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.883149 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.883123 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" Jul 08 18:22:58.988281 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:58.988249 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dqj6" event={"ID":"8652526a-d355-4d28-bbd3-b7a70947c333","Type":"ContainerStarted","Data":"44aeae3aac4bf48e7c401559b0d3864cbc64b3534186d167b2e5b11de058be43"} Jul 08 18:22:59.024164 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.023405 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf"] Jul 08 18:22:59.025950 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:22:59.025926 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bafd724_094e_4ff1_9465_501558427f83.slice/crio-877c249f0c41761ffbe9d2d2591e0e3267a10fa9d2c875d0451e64fb7de8548e WatchSource:0}: Error finding container 877c249f0c41761ffbe9d2d2591e0e3267a10fa9d2c875d0451e64fb7de8548e: Status 404 returned error can't find the container with id 877c249f0c41761ffbe9d2d2591e0e3267a10fa9d2c875d0451e64fb7de8548e Jul 08 18:22:59.063704 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.063229 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:22:59.066957 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.066885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.069468 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.069450 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 18:22:59.070291 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.069839 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 18:22:59.070291 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.070099 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 18:22:59.070291 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.070110 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 18:22:59.070291 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.070104 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 18:22:59.070539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.070317 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 18:22:59.070539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.070417 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ptfz\"" Jul 08 18:22:59.070539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.070532 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 18:22:59.070691 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.070605 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 18:22:59.070884 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.070865 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 18:22:59.083506 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.083480 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:22:59.216710 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.216683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.216885 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.216722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-web-config\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.216885 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.216747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7r4\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-kube-api-access-5l7r4\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.216885 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.216831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217062 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.216878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217062 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.216944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217062 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.217033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.217064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.217100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-out\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.217146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.217232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.217279 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.217441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.217355 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318178 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318338 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318338 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-web-config\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318338 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7r4\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-kube-api-access-5l7r4\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318533 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318533 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318533 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318717 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318717 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318717 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-out\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318717 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318947 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:59.318724 2565 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 08 18:22:59.318947 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318947 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.318760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.318947 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:59.318805 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls podName:a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a nodeName:}" failed. No retries permitted until 2026-07-08 18:22:59.818783461 +0000 UTC m=+314.442370541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a") : secret "alertmanager-main-tls" not found Jul 08 18:22:59.319344 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.319318 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.319595 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.319573 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.319730 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:22:59.319717 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle podName:a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a nodeName:}" failed. No retries permitted until 2026-07-08 18:22:59.819698328 +0000 UTC m=+314.443285390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a") : configmap references non-existent config key: ca-bundle.crt Jul 08 18:22:59.321254 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.321228 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-web-config\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.321374 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.321300 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.321574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.321553 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.322273 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.322246 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.322374 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.322298 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.322475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.322422 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.323414 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.323391 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-out\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.324147 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.324127 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.326872 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.326841 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7r4\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-kube-api-access-5l7r4\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.824450 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.824372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.824450 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.824443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.825373 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.825339 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.827411 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.827390 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.991051 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.991018 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:22:59.995374 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.995347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" event={"ID":"5bafd724-094e-4ff1-9465-501558427f83","Type":"ContainerStarted","Data":"7a8d29bbac7c3c08d4f8c5172707c77f3eaab3e060b1e1f6f6d6fb7b52cae3e1"} Jul 08 18:22:59.995480 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.995408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" event={"ID":"5bafd724-094e-4ff1-9465-501558427f83","Type":"ContainerStarted","Data":"d76e3ff73422ef3964d89782b71fcf8a7decf8c43278c1287995ca9015b9ff22"} Jul 08 18:22:59.995480 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.995429 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" event={"ID":"5bafd724-094e-4ff1-9465-501558427f83","Type":"ContainerStarted","Data":"877c249f0c41761ffbe9d2d2591e0e3267a10fa9d2c875d0451e64fb7de8548e"} Jul 08 18:22:59.997477 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.997447 2565 generic.go:356] "Generic (PLEG): container finished" podID="8652526a-d355-4d28-bbd3-b7a70947c333" containerID="d61a16467cc724020571c42ead22780afec20e9c3483b93b6486de9add751e67" exitCode=0 Jul 08 18:22:59.997579 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:22:59.997534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dqj6" event={"ID":"8652526a-d355-4d28-bbd3-b7a70947c333","Type":"ContainerDied","Data":"d61a16467cc724020571c42ead22780afec20e9c3483b93b6486de9add751e67"} Jul 08 18:23:00.327330 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:00.327305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:23:00.331857 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:23:00.331827 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f8c1d4_7c1c_47ec_afa8_fb66aacc5c1a.slice/crio-85092d92d2edfbb6d971f96506399ec73d111f96cf6ba8a8890adc9ff216fc21 WatchSource:0}: Error finding container 85092d92d2edfbb6d971f96506399ec73d111f96cf6ba8a8890adc9ff216fc21: Status 404 returned error can't find the container with id 85092d92d2edfbb6d971f96506399ec73d111f96cf6ba8a8890adc9ff216fc21 Jul 08 18:23:01.004209 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:01.004172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" event={"ID":"5bafd724-094e-4ff1-9465-501558427f83","Type":"ContainerStarted","Data":"232e5049f9776f917856b831f41bdb13ab68f3e3af3e6c345618f6f05456fb06"} Jul 08 18:23:01.006616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:01.006589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dqj6" event={"ID":"8652526a-d355-4d28-bbd3-b7a70947c333","Type":"ContainerStarted","Data":"68f189b791a2e05137c3b294060f212a0d07d137e27ef05afe07f7d0ba8d8c22"} Jul 08 18:23:01.006760 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:01.006675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dqj6" event={"ID":"8652526a-d355-4d28-bbd3-b7a70947c333","Type":"ContainerStarted","Data":"d7823c5769bbc8b83255c8b688d87666b5ab1c3753e708815e7592de0757b54f"} Jul 08 18:23:01.007941 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:01.007917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerStarted","Data":"85092d92d2edfbb6d971f96506399ec73d111f96cf6ba8a8890adc9ff216fc21"} Jul 08 18:23:01.031116 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:01.031058 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-hnngf" podStartSLOduration=2.9718884660000002 podStartE2EDuration="4.031040547s" totalImagesPullingTime="1.059152081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:23:01.029433494 +0000 UTC m=+315.653020574" watchObservedRunningTime="2026-07-08 18:23:01.031040547 +0000 UTC m=+315.654627628" Jul 08 18:23:01.052836 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:01.052789 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5dqj6" podStartSLOduration=2.762863699 podStartE2EDuration="4.052771886s" totalImagesPullingTime="1.225536983s" totalInitContainerRuntime="64.371204ms" isStatefulPod=true podCreationTimestamp="2026-07-08 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:23:01.05204261 +0000 UTC m=+315.675629692" watchObservedRunningTime="2026-07-08 18:23:01.052771886 +0000 UTC m=+315.676358968" Jul 08 18:23:02.012832 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.012801 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerID="b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e" exitCode=0 Jul 08 18:23:02.013202 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.012891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerDied","Data":"b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e"} Jul 08 18:23:02.592182 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.592151 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c454f47b9-658l2"] Jul 08 18:23:02.754210 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.754173 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f"] Jul 08 18:23:02.757494 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.757478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:02.760171 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.760151 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nw7x7\"" Jul 08 18:23:02.760171 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.760161 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 08 18:23:02.768695 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.767821 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f"] Jul 08 18:23:02.852059 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.851995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ec587e7-85e2-481e-9574-2984b442d150-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9jk6f\" (UID: \"9ec587e7-85e2-481e-9574-2984b442d150\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:02.852167 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.852074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cff8s\" (UniqueName: \"kubernetes.io/projected/9ec587e7-85e2-481e-9574-2984b442d150-kube-api-access-cff8s\") pod \"monitoring-plugin-d5dcff7cf-9jk6f\" (UID: \"9ec587e7-85e2-481e-9574-2984b442d150\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:02.952758 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.952728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cff8s\" (UniqueName: \"kubernetes.io/projected/9ec587e7-85e2-481e-9574-2984b442d150-kube-api-access-cff8s\") pod \"monitoring-plugin-d5dcff7cf-9jk6f\" (UID: \"9ec587e7-85e2-481e-9574-2984b442d150\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:02.952892 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.952822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ec587e7-85e2-481e-9574-2984b442d150-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9jk6f\" (UID: \"9ec587e7-85e2-481e-9574-2984b442d150\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:02.953030 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:23:02.952999 2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 08 18:23:02.953107 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:23:02.953087 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec587e7-85e2-481e-9574-2984b442d150-monitoring-plugin-cert podName:9ec587e7-85e2-481e-9574-2984b442d150 nodeName:}" failed. No retries permitted until 2026-07-08 18:23:03.453066621 +0000 UTC m=+318.076653683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9ec587e7-85e2-481e-9574-2984b442d150-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-9jk6f" (UID: "9ec587e7-85e2-481e-9574-2984b442d150") : secret "monitoring-plugin-cert" not found Jul 08 18:23:02.961933 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:02.961912 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cff8s\" (UniqueName: \"kubernetes.io/projected/9ec587e7-85e2-481e-9574-2984b442d150-kube-api-access-cff8s\") pod \"monitoring-plugin-d5dcff7cf-9jk6f\" (UID: \"9ec587e7-85e2-481e-9574-2984b442d150\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:03.458151 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:03.458110 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ec587e7-85e2-481e-9574-2984b442d150-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9jk6f\" (UID: \"9ec587e7-85e2-481e-9574-2984b442d150\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:03.460935 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:03.460905 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ec587e7-85e2-481e-9574-2984b442d150-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9jk6f\" (UID: \"9ec587e7-85e2-481e-9574-2984b442d150\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:03.667621 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:03.667590 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:03.802730 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:03.802695 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f"] Jul 08 18:23:04.080697 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:23:04.080648 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec587e7_85e2_481e_9574_2984b442d150.slice/crio-1b98853066e36a92b9a63bba6881b4f1dbce4f4c1f58a0c765d2117684de3239 WatchSource:0}: Error finding container 1b98853066e36a92b9a63bba6881b4f1dbce4f4c1f58a0c765d2117684de3239: Status 404 returned error can't find the container with id 1b98853066e36a92b9a63bba6881b4f1dbce4f4c1f58a0c765d2117684de3239 Jul 08 18:23:05.030742 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:05.030704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerStarted","Data":"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c"} Jul 08 18:23:05.031205 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:05.030789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerStarted","Data":"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02"} Jul 08 18:23:05.031205 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:05.030836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerStarted","Data":"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd"} Jul 08 18:23:05.031205 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:05.030890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerStarted","Data":"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6"} Jul 08 18:23:05.031205 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:05.030930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerStarted","Data":"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26"} Jul 08 18:23:05.032225 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:05.032157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" event={"ID":"9ec587e7-85e2-481e-9574-2984b442d150","Type":"ContainerStarted","Data":"1b98853066e36a92b9a63bba6881b4f1dbce4f4c1f58a0c765d2117684de3239"} Jul 08 18:23:06.036744 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:06.036705 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" event={"ID":"9ec587e7-85e2-481e-9574-2984b442d150","Type":"ContainerStarted","Data":"caa1c3e1cff57f80806a4c165da14f91333d25b54658516480336dc016f04d21"} Jul 08 18:23:06.037135 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:06.036768 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:06.040168 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:06.040143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerStarted","Data":"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41"} Jul 08 18:23:06.042160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:06.042136 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" Jul 08 18:23:06.056100 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:06.056054 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9jk6f" podStartSLOduration=2.366167517 podStartE2EDuration="4.056033562s" totalImagesPullingTime="1.689866045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:23:06.052817993 +0000 UTC m=+320.676405074" watchObservedRunningTime="2026-07-08 18:23:06.056033562 +0000 UTC m=+320.679620643" Jul 08 18:23:06.098842 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:06.098803 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.582364515 podStartE2EDuration="7.098788386s" totalImagesPullingTime="4.499146506s" totalInitContainerRuntime="17.277365ms" isStatefulPod=false podCreationTimestamp="2026-07-08 18:22:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:23:06.096998631 +0000 UTC m=+320.720585734" watchObservedRunningTime="2026-07-08 18:23:06.098788386 +0000 UTC m=+320.722375467" Jul 08 18:23:11.926369 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:11.926339 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:23:11.926853 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:23:11.926488 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:23:22.922377 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:22.922344 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:23:22.922899 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:23:22.922528 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:23:27.612350 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.612301 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" podUID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" containerName="registry" containerID="cri-o://5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8" gracePeriod=30 Jul 08 18:23:27.848329 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.848309 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:23:27.934285 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934228 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-certificates\") pod \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " Jul 08 18:23:27.934285 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934260 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09e31dfd-bf6f-446a-af68-bc4c01a1e861-ca-trust-extracted\") pod \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " Jul 08 18:23:27.934484 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934287 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-image-registry-private-configuration\") pod \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " Jul 08 18:23:27.934484 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-installation-pull-secrets\") pod \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " Jul 08 18:23:27.934484 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934335 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") pod \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " Jul 08 18:23:27.934484 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934353 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7g2n\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-kube-api-access-x7g2n\") pod \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " Jul 08 18:23:27.934484 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934406 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-trusted-ca\") pod \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " Jul 08 18:23:27.934770 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934647 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "09e31dfd-bf6f-446a-af68-bc4c01a1e861" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:23:27.934889 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934859 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "09e31dfd-bf6f-446a-af68-bc4c01a1e861" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:23:27.934949 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.934914 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-bound-sa-token\") pod \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\" (UID: \"09e31dfd-bf6f-446a-af68-bc4c01a1e861\") " Jul 08 18:23:27.935207 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.935189 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-trusted-ca\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:23:27.935298 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.935211 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-certificates\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:23:27.936990 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.936894 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "09e31dfd-bf6f-446a-af68-bc4c01a1e861" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:23:27.937180 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.937138 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "09e31dfd-bf6f-446a-af68-bc4c01a1e861" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:23:27.937259 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.937220 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-kube-api-access-x7g2n" (OuterVolumeSpecName: "kube-api-access-x7g2n") pod "09e31dfd-bf6f-446a-af68-bc4c01a1e861" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861"). InnerVolumeSpecName "kube-api-access-x7g2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:23:27.937305 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.937259 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "09e31dfd-bf6f-446a-af68-bc4c01a1e861" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:23:27.937305 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.937298 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "09e31dfd-bf6f-446a-af68-bc4c01a1e861" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:23:27.942360 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:27.942340 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e31dfd-bf6f-446a-af68-bc4c01a1e861-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "09e31dfd-bf6f-446a-af68-bc4c01a1e861" (UID: "09e31dfd-bf6f-446a-af68-bc4c01a1e861"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:23:28.036345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.036318 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7g2n\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-kube-api-access-x7g2n\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:23:28.036451 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.036348 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-bound-sa-token\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:23:28.036451 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.036364 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09e31dfd-bf6f-446a-af68-bc4c01a1e861-ca-trust-extracted\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:23:28.036451 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.036378 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-image-registry-private-configuration\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:23:28.036451 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.036387 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09e31dfd-bf6f-446a-af68-bc4c01a1e861-installation-pull-secrets\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:23:28.036451 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.036400 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09e31dfd-bf6f-446a-af68-bc4c01a1e861-registry-tls\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:23:28.122843 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.122812 2565 generic.go:356] "Generic (PLEG): container finished" podID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" containerID="5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8" exitCode=0 Jul 08 18:23:28.122998 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.122861 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" Jul 08 18:23:28.122998 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.122892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" event={"ID":"09e31dfd-bf6f-446a-af68-bc4c01a1e861","Type":"ContainerDied","Data":"5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8"} Jul 08 18:23:28.122998 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.122937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c454f47b9-658l2" event={"ID":"09e31dfd-bf6f-446a-af68-bc4c01a1e861","Type":"ContainerDied","Data":"95619ee8900dc720e1303e6e2501d666405847e831df8ea1bf63dc7caba2825d"} Jul 08 18:23:28.122998 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.122960 2565 scope.go:117] "RemoveContainer" containerID="5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8" Jul 08 18:23:28.135795 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.135776 2565 scope.go:117] "RemoveContainer" containerID="5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8" Jul 08 18:23:28.136081 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:23:28.136061 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8\": container with ID starting with 5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8 not found: ID does not exist" containerID="5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8" Jul 08 18:23:28.136131 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.136091 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8"} err="failed to get container status \"5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8\": rpc error: code = NotFound desc = could not find container \"5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8\": container with ID starting with 5ac4abdf169d1c698a506910b625bf3ba5aac6f8fb570187965c3a3d3dbc6ca8 not found: ID does not exist" Jul 08 18:23:28.146178 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.146154 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c454f47b9-658l2"] Jul 08 18:23:28.153951 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:28.153931 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c454f47b9-658l2"] Jul 08 18:23:29.926379 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:29.926347 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" path="/var/lib/kubelet/pods/09e31dfd-bf6f-446a-af68-bc4c01a1e861/volumes" Jul 08 18:23:34.921504 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:34.921475 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:23:34.921872 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:23:34.921611 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:23:47.192809 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:47.192746 2565 generic.go:356] "Generic (PLEG): container finished" podID="38444fe4-2b88-467a-bd50-8183c175e6ed" containerID="3a25ce25c1678104b60322eb97f4cedb12373d534e190571d443a9b1eb12133a" exitCode=0 Jul 08 18:23:47.193131 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:47.192816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" event={"ID":"38444fe4-2b88-467a-bd50-8183c175e6ed","Type":"ContainerDied","Data":"3a25ce25c1678104b60322eb97f4cedb12373d534e190571d443a9b1eb12133a"} Jul 08 18:23:47.193131 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:47.193120 2565 scope.go:117] "RemoveContainer" containerID="3a25ce25c1678104b60322eb97f4cedb12373d534e190571d443a9b1eb12133a" Jul 08 18:23:47.922390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:47.922362 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:23:47.922538 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:23:47.922510 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:23:48.198307 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:48.198228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4dpmn" event={"ID":"38444fe4-2b88-467a-bd50-8183c175e6ed","Type":"ContainerStarted","Data":"84c22fb0e09f284539db722d2c1109b2aa88cdf0ed78bf2dc9fe71399dc6b4d8"} Jul 08 18:23:58.921932 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:58.921901 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:23:58.922361 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:23:58.922075 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-68b757fb67-g44vm_openshift-console-operator(fac2c03c-dff8-4df8-8400-0e30266cabfa)\"" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podUID="fac2c03c-dff8-4df8-8400-0e30266cabfa" Jul 08 18:23:59.659791 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:59.659763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:23:59.661941 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:59.661923 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fe673d-c3eb-4eda-8558-6498fbd55bd7-metrics-certs\") pod \"network-metrics-daemon-4dl6h\" (UID: \"79fe673d-c3eb-4eda-8558-6498fbd55bd7\") " pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:23:59.724912 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:59.724891 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9qsc4\"" Jul 08 18:23:59.732174 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:59.732157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4dl6h" Jul 08 18:23:59.854857 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:23:59.854766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4dl6h"] Jul 08 18:23:59.857319 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:23:59.857290 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fe673d_c3eb_4eda_8558_6498fbd55bd7.slice/crio-589d01dc05efa19b96b3fc70ee0d4c42b8156da5307e17ea034146e051bd2ed9 WatchSource:0}: Error finding container 589d01dc05efa19b96b3fc70ee0d4c42b8156da5307e17ea034146e051bd2ed9: Status 404 returned error can't find the container with id 589d01dc05efa19b96b3fc70ee0d4c42b8156da5307e17ea034146e051bd2ed9 Jul 08 18:24:00.242054 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:00.242025 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4dl6h" event={"ID":"79fe673d-c3eb-4eda-8558-6498fbd55bd7","Type":"ContainerStarted","Data":"589d01dc05efa19b96b3fc70ee0d4c42b8156da5307e17ea034146e051bd2ed9"} Jul 08 18:24:01.247701 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:01.247649 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4dl6h" event={"ID":"79fe673d-c3eb-4eda-8558-6498fbd55bd7","Type":"ContainerStarted","Data":"8af264062e4be872de4324771bb7437a5ac715be3b6ef75fe320d18196dc7e08"} Jul 08 18:24:01.248039 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:01.247724 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4dl6h" event={"ID":"79fe673d-c3eb-4eda-8558-6498fbd55bd7","Type":"ContainerStarted","Data":"d191bf8cab2851eb820c8bb9c88be88a90aeb3458f45eb2432a1043dadc796f8"} Jul 08 18:24:01.265132 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:01.265095 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4dl6h" podStartSLOduration=374.257159735 podStartE2EDuration="6m15.265080236s" totalImagesPullingTime="1.007920501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:24:01.264090573 +0000 UTC m=+375.887677655" watchObservedRunningTime="2026-07-08 18:24:01.265080236 +0000 UTC m=+375.888667313" Jul 08 18:24:10.922259 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:10.922232 2565 scope.go:117] "RemoveContainer" containerID="b8f3b0571d020ad063bb1f9c664e44a08a10cc430aae590b449b731cf3799610" Jul 08 18:24:11.285977 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.285953 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:24:11.286140 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.286078 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" event={"ID":"fac2c03c-dff8-4df8-8400-0e30266cabfa","Type":"ContainerStarted","Data":"5eb682c4d377bc059db99ab2458a4bcb320a4f8262a46640bc8d7e3d691d4c56"} Jul 08 18:24:11.286378 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.286356 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:24:11.305011 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.304949 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" podStartSLOduration=378.479331424 podStartE2EDuration="6m22.304935105s" totalImagesPullingTime="3.825603681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:24:11.303462446 +0000 UTC m=+385.927049553" watchObservedRunningTime="2026-07-08 18:24:11.304935105 +0000 UTC m=+385.928522222" Jul 08 18:24:11.585204 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.585148 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-g44vm" Jul 08 18:24:11.776003 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.775973 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-xzrfz"] Jul 08 18:24:11.776339 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.776325 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" containerName="registry" Jul 08 18:24:11.776385 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.776341 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" containerName="registry" Jul 08 18:24:11.776418 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.776399 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e31dfd-bf6f-446a-af68-bc4c01a1e861" containerName="registry" Jul 08 18:24:11.778175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.778158 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-xzrfz" Jul 08 18:24:11.780640 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.780616 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 18:24:11.780743 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.780626 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 18:24:11.780743 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.780692 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mbdws\"" Jul 08 18:24:11.788493 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.788476 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-xzrfz"] Jul 08 18:24:11.857636 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.857587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp4lv\" (UniqueName: \"kubernetes.io/projected/a1041e9a-201a-4897-83f8-afe6f64c3cf2-kube-api-access-hp4lv\") pod \"downloads-685bfb5f64-xzrfz\" (UID: \"a1041e9a-201a-4897-83f8-afe6f64c3cf2\") " pod="openshift-console/downloads-685bfb5f64-xzrfz" Jul 08 18:24:11.958889 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.958871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp4lv\" (UniqueName: \"kubernetes.io/projected/a1041e9a-201a-4897-83f8-afe6f64c3cf2-kube-api-access-hp4lv\") pod \"downloads-685bfb5f64-xzrfz\" (UID: \"a1041e9a-201a-4897-83f8-afe6f64c3cf2\") " pod="openshift-console/downloads-685bfb5f64-xzrfz" Jul 08 18:24:11.967674 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:11.967648 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp4lv\" (UniqueName: \"kubernetes.io/projected/a1041e9a-201a-4897-83f8-afe6f64c3cf2-kube-api-access-hp4lv\") pod \"downloads-685bfb5f64-xzrfz\" (UID: \"a1041e9a-201a-4897-83f8-afe6f64c3cf2\") " pod="openshift-console/downloads-685bfb5f64-xzrfz" Jul 08 18:24:12.087634 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:12.087614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-xzrfz" Jul 08 18:24:12.211602 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:12.211580 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-xzrfz"] Jul 08 18:24:12.213346 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:24:12.213316 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1041e9a_201a_4897_83f8_afe6f64c3cf2.slice/crio-a21f43212a31547f493f4f3d11b6a94e59e1bebfd54845aa6417ca3a59a47f8f WatchSource:0}: Error finding container a21f43212a31547f493f4f3d11b6a94e59e1bebfd54845aa6417ca3a59a47f8f: Status 404 returned error can't find the container with id a21f43212a31547f493f4f3d11b6a94e59e1bebfd54845aa6417ca3a59a47f8f Jul 08 18:24:12.290543 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:12.290516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-xzrfz" event={"ID":"a1041e9a-201a-4897-83f8-afe6f64c3cf2","Type":"ContainerStarted","Data":"a21f43212a31547f493f4f3d11b6a94e59e1bebfd54845aa6417ca3a59a47f8f"} Jul 08 18:24:21.201916 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.201885 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-564949d647-sh6wf"] Jul 08 18:24:21.204007 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.203988 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.207642 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.207587 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 18:24:21.207642 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.207595 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 18:24:21.207642 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.207616 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 18:24:21.207917 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.207594 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 18:24:21.207917 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.207607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 18:24:21.207917 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.207587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mjj2n\"" Jul 08 18:24:21.217358 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.217334 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564949d647-sh6wf"] Jul 08 18:24:21.336693 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.336653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-service-ca\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.336840 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.336729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-serving-cert\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.336840 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.336768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-oauth-serving-cert\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.336840 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.336803 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/41e91d08-7c69-4959-ac36-fe5a10097787-kube-api-access-4wq7c\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.336962 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.336876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-oauth-config\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.336962 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.336900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-console-config\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.437486 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.437461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-oauth-config\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.437605 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.437490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-console-config\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.437605 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.437523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-service-ca\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.437605 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.437566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-serving-cert\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.437605 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.437588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-oauth-serving-cert\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.437842 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.437609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/41e91d08-7c69-4959-ac36-fe5a10097787-kube-api-access-4wq7c\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.438278 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.438218 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-console-config\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.438394 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.438328 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-oauth-serving-cert\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.438394 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.438385 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-service-ca\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.440062 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.440042 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-serving-cert\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.440173 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.440153 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-oauth-config\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.445630 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.445612 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/41e91d08-7c69-4959-ac36-fe5a10097787-kube-api-access-4wq7c\") pod \"console-564949d647-sh6wf\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.514242 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.514223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:21.658951 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:21.658929 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564949d647-sh6wf"] Jul 08 18:24:21.661588 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:24:21.661549 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e91d08_7c69_4959_ac36_fe5a10097787.slice/crio-a7d3f4a1248c65eac0cf0bbc7fdbf566c276bbd26908de59620255cd79232f68 WatchSource:0}: Error finding container a7d3f4a1248c65eac0cf0bbc7fdbf566c276bbd26908de59620255cd79232f68: Status 404 returned error can't find the container with id a7d3f4a1248c65eac0cf0bbc7fdbf566c276bbd26908de59620255cd79232f68 Jul 08 18:24:22.334898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:22.334857 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564949d647-sh6wf" event={"ID":"41e91d08-7c69-4959-ac36-fe5a10097787","Type":"ContainerStarted","Data":"a7d3f4a1248c65eac0cf0bbc7fdbf566c276bbd26908de59620255cd79232f68"} Jul 08 18:24:25.350472 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:25.350433 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564949d647-sh6wf" event={"ID":"41e91d08-7c69-4959-ac36-fe5a10097787","Type":"ContainerStarted","Data":"5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732"} Jul 08 18:24:25.369105 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:25.369053 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-564949d647-sh6wf" podStartSLOduration=1.460484325 podStartE2EDuration="4.369037144s" totalImagesPullingTime="2.908552819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:24:25.368041034 +0000 UTC m=+399.991628117" watchObservedRunningTime="2026-07-08 18:24:25.369037144 +0000 UTC m=+399.992624225" Jul 08 18:24:28.662350 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.662312 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75ff4d5fd7-4bfg9"] Jul 08 18:24:28.665239 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.665216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.673787 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.673762 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 18:24:28.677006 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.676979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75ff4d5fd7-4bfg9"] Jul 08 18:24:28.702392 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.702362 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-trusted-ca-bundle\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.702521 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.702419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-config\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.702521 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.702487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-service-ca\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.702521 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.702512 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9npf\" (UniqueName: \"kubernetes.io/projected/2c292b26-5946-4dd9-87fa-2c59b000e2ec-kube-api-access-h9npf\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.702689 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.702547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-oauth-config\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.702689 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.702614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-serving-cert\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.702689 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.702650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-oauth-serving-cert\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.803993 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.803961 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-service-ca\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.804165 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9npf\" (UniqueName: \"kubernetes.io/projected/2c292b26-5946-4dd9-87fa-2c59b000e2ec-kube-api-access-h9npf\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.804165 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-oauth-config\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.804165 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-serving-cert\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.804165 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-oauth-serving-cert\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.804165 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-trusted-ca-bundle\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.804414 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-config\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.804881 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804853 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-service-ca\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.805010 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804858 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-oauth-serving-cert\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.805010 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.804969 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-config\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.805230 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.805199 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c292b26-5946-4dd9-87fa-2c59b000e2ec-trusted-ca-bundle\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.806942 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.806917 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-serving-cert\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.807081 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.807061 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c292b26-5946-4dd9-87fa-2c59b000e2ec-console-oauth-config\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.812966 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.812939 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9npf\" (UniqueName: \"kubernetes.io/projected/2c292b26-5946-4dd9-87fa-2c59b000e2ec-kube-api-access-h9npf\") pod \"console-75ff4d5fd7-4bfg9\" (UID: \"2c292b26-5946-4dd9-87fa-2c59b000e2ec\") " pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:28.977580 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:28.977493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:29.842318 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:24:29.842279 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" podUID="a6e10b2d-d75a-41ba-993e-35bfe3217286" Jul 08 18:24:30.375307 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:30.375278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:24:31.514503 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:31.514468 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:31.514937 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:31.514577 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:31.519925 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:31.519900 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:31.727582 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:31.727542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:24:31.730317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:31.730295 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e10b2d-d75a-41ba-993e-35bfe3217286-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xlmdv\" (UID: \"a6e10b2d-d75a-41ba-993e-35bfe3217286\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:24:31.878867 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:31.878796 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-n4nnk\"" Jul 08 18:24:31.886594 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:31.886572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" Jul 08 18:24:32.388380 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:32.388351 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:24:33.117577 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:33.117551 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75ff4d5fd7-4bfg9"] Jul 08 18:24:33.120051 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:24:33.120025 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c292b26_5946_4dd9_87fa_2c59b000e2ec.slice/crio-97b0a1b0f80840a7f70f4cb405cad94ef814a75b0497c200fe57bddf26b3e273 WatchSource:0}: Error finding container 97b0a1b0f80840a7f70f4cb405cad94ef814a75b0497c200fe57bddf26b3e273: Status 404 returned error can't find the container with id 97b0a1b0f80840a7f70f4cb405cad94ef814a75b0497c200fe57bddf26b3e273 Jul 08 18:24:33.143430 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:33.143409 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv"] Jul 08 18:24:33.145507 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:24:33.145485 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e10b2d_d75a_41ba_993e_35bfe3217286.slice/crio-fe4864db23e82f4730a4e6e88cf9d5feb32e0c319ec5aa26b46ec42389925f99 WatchSource:0}: Error finding container fe4864db23e82f4730a4e6e88cf9d5feb32e0c319ec5aa26b46ec42389925f99: Status 404 returned error can't find the container with id fe4864db23e82f4730a4e6e88cf9d5feb32e0c319ec5aa26b46ec42389925f99 Jul 08 18:24:33.390708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:33.390676 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" event={"ID":"a6e10b2d-d75a-41ba-993e-35bfe3217286","Type":"ContainerStarted","Data":"fe4864db23e82f4730a4e6e88cf9d5feb32e0c319ec5aa26b46ec42389925f99"} Jul 08 18:24:33.392747 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:33.392707 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ff4d5fd7-4bfg9" event={"ID":"2c292b26-5946-4dd9-87fa-2c59b000e2ec","Type":"ContainerStarted","Data":"6e4f4928d672d780b21b54fa68a114489acc96c30a785e7aa213dfb22e9a898b"} Jul 08 18:24:33.392747 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:33.392758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ff4d5fd7-4bfg9" event={"ID":"2c292b26-5946-4dd9-87fa-2c59b000e2ec","Type":"ContainerStarted","Data":"97b0a1b0f80840a7f70f4cb405cad94ef814a75b0497c200fe57bddf26b3e273"} Jul 08 18:24:33.394591 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:33.394560 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-xzrfz" event={"ID":"a1041e9a-201a-4897-83f8-afe6f64c3cf2","Type":"ContainerStarted","Data":"92b4e8188b935cfa4649ed5ef17e3be1051ec8fa5a2cf57143700ce70b0e4847"} Jul 08 18:24:33.414620 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:33.414566 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75ff4d5fd7-4bfg9" podStartSLOduration=5.414537778 podStartE2EDuration="5.414537778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:24:33.411907108 +0000 UTC m=+408.035494190" watchObservedRunningTime="2026-07-08 18:24:33.414537778 +0000 UTC m=+408.038124862" Jul 08 18:24:33.430560 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:33.430518 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-xzrfz" podStartSLOduration=1.5473314889999998 podStartE2EDuration="22.430505611s" totalImagesPullingTime="20.883174122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:24:33.428549441 +0000 UTC m=+408.052136522" watchObservedRunningTime="2026-07-08 18:24:33.430505611 +0000 UTC m=+408.054092690" Jul 08 18:24:34.399824 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:34.399787 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-xzrfz" Jul 08 18:24:34.418388 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:34.418357 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-xzrfz" Jul 08 18:24:35.406372 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:35.406280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" event={"ID":"a6e10b2d-d75a-41ba-993e-35bfe3217286","Type":"ContainerStarted","Data":"ea3625d5326396f7da43c7689fe495962eebdf543076c6e3993829228f0089da"} Jul 08 18:24:35.423898 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:35.423850 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xlmdv" podStartSLOduration=381.494223473 podStartE2EDuration="6m23.423834608s" totalImagesPullingTime="1.929611135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:24:35.422453859 +0000 UTC m=+410.046040968" watchObservedRunningTime="2026-07-08 18:24:35.423834608 +0000 UTC m=+410.047421716" Jul 08 18:24:36.673629 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:36.673590 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:24:36.674177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:36.673939 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="alertmanager" containerID="cri-o://d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26" gracePeriod=120 Jul 08 18:24:36.674177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:36.673954 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="prom-label-proxy" containerID="cri-o://fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41" gracePeriod=120 Jul 08 18:24:36.674177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:36.673982 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy" containerID="cri-o://648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02" gracePeriod=120 Jul 08 18:24:36.674177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:36.674020 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="config-reloader" containerID="cri-o://3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6" gracePeriod=120 Jul 08 18:24:36.674177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:36.673992 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy-web" containerID="cri-o://cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd" gracePeriod=120 Jul 08 18:24:36.674177 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:36.674128 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy-metric" containerID="cri-o://c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c" gracePeriod=120 Jul 08 18:24:37.419593 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.419552 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerID="fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41" exitCode=0 Jul 08 18:24:37.419593 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.419584 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerID="648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02" exitCode=0 Jul 08 18:24:37.419593 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.419591 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerID="3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6" exitCode=0 Jul 08 18:24:37.419593 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.419598 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerID="d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26" exitCode=0 Jul 08 18:24:37.419888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.419628 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerDied","Data":"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41"} Jul 08 18:24:37.419888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.419730 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerDied","Data":"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02"} Jul 08 18:24:37.419888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.419780 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerDied","Data":"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6"} Jul 08 18:24:37.419888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.419820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerDied","Data":"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26"} Jul 08 18:24:37.978931 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:37.978903 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.096461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096379 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-volume\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096457 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096692 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096577 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-web-config\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096692 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096622 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096692 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096655 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-main-db\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096872 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096712 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096872 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096766 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-web\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096872 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096797 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l7r4\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-kube-api-access-5l7r4\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096872 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096831 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-metrics-client-ca\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.096872 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096855 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.097104 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096882 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-tls-assets\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.097104 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096909 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-cluster-tls-config\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.097104 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.096968 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-out\") pod \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\" (UID: \"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a\") " Jul 08 18:24:38.097104 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.097001 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:24:38.097279 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.097223 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:24:38.097279 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.097236 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.098288 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.097472 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:24:38.099337 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.099303 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-volume" (OuterVolumeSpecName: "config-volume") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:24:38.100410 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.100379 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:24:38.101449 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.101409 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:24:38.101690 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.101636 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:24:38.101862 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.101823 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-out" (OuterVolumeSpecName: "config-out") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:24:38.102157 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.102124 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-kube-api-access-5l7r4" (OuterVolumeSpecName: "kube-api-access-5l7r4") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "kube-api-access-5l7r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:24:38.102323 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.102296 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:24:38.103218 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.103183 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:24:38.107280 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.107208 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:24:38.115097 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.115076 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-web-config" (OuterVolumeSpecName: "web-config") pod "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" (UID: "a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:24:38.198395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198357 2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-volume\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198391 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198408 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-web-config\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198420 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-alertmanager-main-db\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198433 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-main-tls\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198446 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198456 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l7r4\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-kube-api-access-5l7r4\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198464 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-metrics-client-ca\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198473 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198482 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-tls-assets\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198496 2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-cluster-tls-config\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.198611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.198503 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a-config-out\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:24:38.428679 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.428587 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerID="c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c" exitCode=0 Jul 08 18:24:38.428679 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.428617 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerID="cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd" exitCode=0 Jul 08 18:24:38.428888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.428655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerDied","Data":"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c"} Jul 08 18:24:38.428888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.428721 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.428888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.428768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerDied","Data":"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd"} Jul 08 18:24:38.428888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.428813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a","Type":"ContainerDied","Data":"85092d92d2edfbb6d971f96506399ec73d111f96cf6ba8a8890adc9ff216fc21"} Jul 08 18:24:38.428888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.428840 2565 scope.go:117] "RemoveContainer" containerID="fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41" Jul 08 18:24:38.444819 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.444794 2565 scope.go:117] "RemoveContainer" containerID="c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c" Jul 08 18:24:38.455146 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.455125 2565 scope.go:117] "RemoveContainer" containerID="648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02" Jul 08 18:24:38.458057 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.458020 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:24:38.464622 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.464605 2565 scope.go:117] "RemoveContainer" containerID="cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd" Jul 08 18:24:38.464933 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.464909 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:24:38.472539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.472521 2565 scope.go:117] "RemoveContainer" containerID="3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6" Jul 08 18:24:38.480401 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.480374 2565 scope.go:117] "RemoveContainer" containerID="d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26" Jul 08 18:24:38.488352 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.488330 2565 scope.go:117] "RemoveContainer" containerID="b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e" Jul 08 18:24:38.494403 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494350 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:24:38.494810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494792 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy" Jul 08 18:24:38.494810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494811 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494823 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="prom-label-proxy" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494831 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="prom-label-proxy" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494846 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="config-reloader" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494854 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="config-reloader" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494865 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="init-config-reloader" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494873 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="init-config-reloader" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494882 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy-metric" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494890 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy-metric" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494915 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="alertmanager" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494923 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="alertmanager" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494936 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy-web" Jul 08 18:24:38.494973 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.494942 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy-web" Jul 08 18:24:38.495422 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.495014 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="alertmanager" Jul 08 18:24:38.495422 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.495026 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy-metric" Jul 08 18:24:38.495422 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.495036 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy-web" Jul 08 18:24:38.495422 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.495044 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="kube-rbac-proxy" Jul 08 18:24:38.495422 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.495055 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="config-reloader" Jul 08 18:24:38.495422 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.495061 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" containerName="prom-label-proxy" Jul 08 18:24:38.496967 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.496935 2565 scope.go:117] "RemoveContainer" containerID="fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41" Jul 08 18:24:38.497305 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:24:38.497255 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41\": container with ID starting with fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41 not found: ID does not exist" containerID="fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41" Jul 08 18:24:38.497391 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.497316 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41"} err="failed to get container status \"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41\": rpc error: code = NotFound desc = could not find container \"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41\": container with ID starting with fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41 not found: ID does not exist" Jul 08 18:24:38.497391 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.497344 2565 scope.go:117] "RemoveContainer" containerID="c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c" Jul 08 18:24:38.497649 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:24:38.497628 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c\": container with ID starting with c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c not found: ID does not exist" containerID="c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c" Jul 08 18:24:38.497758 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.497720 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c"} err="failed to get container status \"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c\": rpc error: code = NotFound desc = could not find container \"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c\": container with ID starting with c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c not found: ID does not exist" Jul 08 18:24:38.497758 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.497745 2565 scope.go:117] "RemoveContainer" containerID="648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02" Jul 08 18:24:38.498024 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:24:38.498006 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02\": container with ID starting with 648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02 not found: ID does not exist" containerID="648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02" Jul 08 18:24:38.498089 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.498031 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02"} err="failed to get container status \"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02\": rpc error: code = NotFound desc = could not find container \"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02\": container with ID starting with 648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02 not found: ID does not exist" Jul 08 18:24:38.498089 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.498052 2565 scope.go:117] "RemoveContainer" containerID="cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd" Jul 08 18:24:38.498338 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:24:38.498315 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd\": container with ID starting with cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd not found: ID does not exist" containerID="cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd" Jul 08 18:24:38.498432 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.498343 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd"} err="failed to get container status \"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd\": rpc error: code = NotFound desc = could not find container \"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd\": container with ID starting with cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd not found: ID does not exist" Jul 08 18:24:38.498432 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.498374 2565 scope.go:117] "RemoveContainer" containerID="3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6" Jul 08 18:24:38.498627 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:24:38.498605 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6\": container with ID starting with 3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6 not found: ID does not exist" containerID="3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6" Jul 08 18:24:38.498721 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.498633 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6"} err="failed to get container status \"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6\": rpc error: code = NotFound desc = could not find container \"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6\": container with ID starting with 3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6 not found: ID does not exist" Jul 08 18:24:38.498721 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.498651 2565 scope.go:117] "RemoveContainer" containerID="d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26" Jul 08 18:24:38.498948 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:24:38.498929 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26\": container with ID starting with d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26 not found: ID does not exist" containerID="d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26" Jul 08 18:24:38.499032 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.498963 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26"} err="failed to get container status \"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26\": rpc error: code = NotFound desc = could not find container \"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26\": container with ID starting with d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26 not found: ID does not exist" Jul 08 18:24:38.499032 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.498982 2565 scope.go:117] "RemoveContainer" containerID="b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e" Jul 08 18:24:38.499224 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:24:38.499203 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e\": container with ID starting with b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e not found: ID does not exist" containerID="b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e" Jul 08 18:24:38.499315 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.499224 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e"} err="failed to get container status \"b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e\": rpc error: code = NotFound desc = could not find container \"b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e\": container with ID starting with b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e not found: ID does not exist" Jul 08 18:24:38.499315 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.499237 2565 scope.go:117] "RemoveContainer" containerID="fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41" Jul 08 18:24:38.499511 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.499482 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41"} err="failed to get container status \"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41\": rpc error: code = NotFound desc = could not find container \"fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41\": container with ID starting with fd97054c3be3dcdf48e642a1043e372bd4d4176b71333b178596097837b42f41 not found: ID does not exist" Jul 08 18:24:38.499618 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.499513 2565 scope.go:117] "RemoveContainer" containerID="c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c" Jul 08 18:24:38.499818 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.499799 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c"} err="failed to get container status \"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c\": rpc error: code = NotFound desc = could not find container \"c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c\": container with ID starting with c024d390e26b2ff164101dd0c0461982b4435568897a47906e95521c92fc111c not found: ID does not exist" Jul 08 18:24:38.499818 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.499818 2565 scope.go:117] "RemoveContainer" containerID="648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02" Jul 08 18:24:38.500095 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.500068 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02"} err="failed to get container status \"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02\": rpc error: code = NotFound desc = could not find container \"648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02\": container with ID starting with 648a5c1c9adabb1bd2facd4abc42f55e77df5ba2733c01d829f9bf07c26cfa02 not found: ID does not exist" Jul 08 18:24:38.500095 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.500093 2565 scope.go:117] "RemoveContainer" containerID="cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd" Jul 08 18:24:38.500343 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.500326 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd"} err="failed to get container status \"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd\": rpc error: code = NotFound desc = could not find container \"cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd\": container with ID starting with cc24fd0fadb3c10efa16ec1417a2c322b2bac88f59f31d3963ca4b6192c092bd not found: ID does not exist" Jul 08 18:24:38.500343 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.500343 2565 scope.go:117] "RemoveContainer" containerID="3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6" Jul 08 18:24:38.500586 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.500566 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6"} err="failed to get container status \"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6\": rpc error: code = NotFound desc = could not find container \"3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6\": container with ID starting with 3e6cd1693e504777660254876f833873dae5d6769b6ee083096f9550766948b6 not found: ID does not exist" Jul 08 18:24:38.500681 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.500588 2565 scope.go:117] "RemoveContainer" containerID="d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26" Jul 08 18:24:38.500840 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.500820 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26"} err="failed to get container status \"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26\": rpc error: code = NotFound desc = could not find container \"d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26\": container with ID starting with d8feb879935af1ede5840f5c3b4f1329cd3a9a802d08fa6dde622057df786e26 not found: ID does not exist" Jul 08 18:24:38.500913 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.500841 2565 scope.go:117] "RemoveContainer" containerID="b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e" Jul 08 18:24:38.501110 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.501088 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e"} err="failed to get container status \"b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e\": rpc error: code = NotFound desc = could not find container \"b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e\": container with ID starting with b4aa76b2bc137d0bb7468f6fbef02a1e68a2ec7b8ad6401c58d0fe0374c31e5e not found: ID does not exist" Jul 08 18:24:38.528235 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.528196 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:24:38.528362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.528344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.530978 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.530943 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ptfz\"" Jul 08 18:24:38.531082 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.530942 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 18:24:38.531082 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.530942 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 18:24:38.531450 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.531429 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 18:24:38.532110 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.531791 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 18:24:38.532110 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.531830 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 18:24:38.532110 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.531848 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 18:24:38.532110 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.531797 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 18:24:38.532365 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.532139 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 18:24:38.536315 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.536294 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 18:24:38.602219 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602363 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8h4t\" (UniqueName: \"kubernetes.io/projected/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-kube-api-access-z8h4t\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602363 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-config-out\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602363 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602587 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602395 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602587 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602448 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602587 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602528 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602587 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602565 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-web-config\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.602972 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.602814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.703857 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.703779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h4t\" (UniqueName: \"kubernetes.io/projected/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-kube-api-access-z8h4t\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.703857 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.703827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-config-out\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704070 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.703863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704070 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.703911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704070 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.703953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704070 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.703989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704070 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.704013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704070 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.704051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.704108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.704134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.704161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-web-config\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.704186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.704210 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.704461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.704356 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.705174 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.705146 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.705384 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.705361 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.706833 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.706802 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-config-out\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.707132 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.707103 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.707233 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.707200 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.707862 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.707838 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-web-config\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.708045 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.708023 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.709000 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.708977 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.709248 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.709216 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.709382 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.709316 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.709707 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.709690 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.712482 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.712459 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8h4t\" (UniqueName: \"kubernetes.io/projected/cf4f22f0-1d09-4f37-80c5-bddd43e48f5d-kube-api-access-z8h4t\") pod \"alertmanager-main-0\" (UID: \"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.840169 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.840131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 18:24:38.978100 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.978070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:38.978504 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.978456 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:38.984325 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:38.984299 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:39.022997 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:39.022970 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 18:24:39.025911 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:24:39.025885 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4f22f0_1d09_4f37_80c5_bddd43e48f5d.slice/crio-d81bf2802e75ba4452df8690c0c7a992b12729bdddba2d7d38012f701bcfdfa2 WatchSource:0}: Error finding container d81bf2802e75ba4452df8690c0c7a992b12729bdddba2d7d38012f701bcfdfa2: Status 404 returned error can't find the container with id d81bf2802e75ba4452df8690c0c7a992b12729bdddba2d7d38012f701bcfdfa2 Jul 08 18:24:39.437193 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:39.437156 2565 generic.go:356] "Generic (PLEG): container finished" podID="cf4f22f0-1d09-4f37-80c5-bddd43e48f5d" containerID="30018321ec8be12a6bc27fdf73a29ac310f879ba4c221cf04d78f38b3f69210f" exitCode=0 Jul 08 18:24:39.437412 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:39.437248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d","Type":"ContainerDied","Data":"30018321ec8be12a6bc27fdf73a29ac310f879ba4c221cf04d78f38b3f69210f"} Jul 08 18:24:39.437412 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:39.437345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d","Type":"ContainerStarted","Data":"d81bf2802e75ba4452df8690c0c7a992b12729bdddba2d7d38012f701bcfdfa2"} Jul 08 18:24:39.441866 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:39.441846 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75ff4d5fd7-4bfg9" Jul 08 18:24:39.512826 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:39.512799 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564949d647-sh6wf"] Jul 08 18:24:39.929573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:39.929545 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a" path="/var/lib/kubelet/pods/a2f8c1d4-7c1c-47ec-afa8-fb66aacc5c1a/volumes" Jul 08 18:24:40.449189 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:40.449148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d","Type":"ContainerStarted","Data":"ace37837eaf3cb5301e642907d8e209d77b3de3798db08183c5ab91b701aac9b"} Jul 08 18:24:40.449714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:40.449267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d","Type":"ContainerStarted","Data":"25aa97acdb6908b6976a70057f616e4aab8f6532eb9db2456f4ca78a589af32a"} Jul 08 18:24:40.449714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:40.449313 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d","Type":"ContainerStarted","Data":"c67e0fae7afbb84c8da5f76803b722d117e38e4a329eeaa9e2a9b162ea868c54"} Jul 08 18:24:40.449714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:40.449341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d","Type":"ContainerStarted","Data":"86df08c23578e8d9447bb71148359a5ed9a00649821b42ecc8148423bb6ef644"} Jul 08 18:24:40.449714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:40.449366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d","Type":"ContainerStarted","Data":"fd552d7f4966aca3b8a7b43e984bf3b02157c3b382a7f34b0f4e0ce3da734dde"} Jul 08 18:24:40.449714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:40.449403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf4f22f0-1d09-4f37-80c5-bddd43e48f5d","Type":"ContainerStarted","Data":"34e7ce9b983b30f586d249b77368246b99a7669e4363e86732252291b4975e77"} Jul 08 18:24:40.477469 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:24:40.477424 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.451433826 podStartE2EDuration="2.477408361s" totalImagesPullingTime="0s" totalInitContainerRuntime="25.974535ms" isStatefulPod=false podCreationTimestamp="2026-07-08 18:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:24:40.474766847 +0000 UTC m=+415.098353950" watchObservedRunningTime="2026-07-08 18:24:40.477408361 +0000 UTC m=+415.100995444" Jul 08 18:25:04.543200 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.543158 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-564949d647-sh6wf" podUID="41e91d08-7c69-4959-ac36-fe5a10097787" containerName="console" containerID="cri-o://5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732" gracePeriod=15 Jul 08 18:25:04.816198 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.816146 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:25:04.908323 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908301 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-console-config\") pod \"41e91d08-7c69-4959-ac36-fe5a10097787\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " Jul 08 18:25:04.908449 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908340 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-service-ca\") pod \"41e91d08-7c69-4959-ac36-fe5a10097787\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " Jul 08 18:25:04.908449 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908372 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-serving-cert\") pod \"41e91d08-7c69-4959-ac36-fe5a10097787\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " Jul 08 18:25:04.908449 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908438 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-oauth-config\") pod \"41e91d08-7c69-4959-ac36-fe5a10097787\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " Jul 08 18:25:04.908549 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908480 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-oauth-serving-cert\") pod \"41e91d08-7c69-4959-ac36-fe5a10097787\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " Jul 08 18:25:04.908549 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908505 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/41e91d08-7c69-4959-ac36-fe5a10097787-kube-api-access-4wq7c\") pod \"41e91d08-7c69-4959-ac36-fe5a10097787\" (UID: \"41e91d08-7c69-4959-ac36-fe5a10097787\") " Jul 08 18:25:04.908976 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908726 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-service-ca" (OuterVolumeSpecName: "service-ca") pod "41e91d08-7c69-4959-ac36-fe5a10097787" (UID: "41e91d08-7c69-4959-ac36-fe5a10097787"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:25:04.908976 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908739 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-console-config" (OuterVolumeSpecName: "console-config") pod "41e91d08-7c69-4959-ac36-fe5a10097787" (UID: "41e91d08-7c69-4959-ac36-fe5a10097787"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:25:04.908976 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.908916 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "41e91d08-7c69-4959-ac36-fe5a10097787" (UID: "41e91d08-7c69-4959-ac36-fe5a10097787"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:25:04.910694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.910648 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "41e91d08-7c69-4959-ac36-fe5a10097787" (UID: "41e91d08-7c69-4959-ac36-fe5a10097787"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:25:04.910694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.910681 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "41e91d08-7c69-4959-ac36-fe5a10097787" (UID: "41e91d08-7c69-4959-ac36-fe5a10097787"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:25:04.911076 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:04.911053 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e91d08-7c69-4959-ac36-fe5a10097787-kube-api-access-4wq7c" (OuterVolumeSpecName: "kube-api-access-4wq7c") pod "41e91d08-7c69-4959-ac36-fe5a10097787" (UID: "41e91d08-7c69-4959-ac36-fe5a10097787"). InnerVolumeSpecName "kube-api-access-4wq7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:25:05.009887 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.009863 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-console-config\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:05.009887 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.009883 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-service-ca\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:05.010005 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.009893 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-serving-cert\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:05.010005 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.009901 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41e91d08-7c69-4959-ac36-fe5a10097787-console-oauth-config\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:05.010005 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.009910 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41e91d08-7c69-4959-ac36-fe5a10097787-oauth-serving-cert\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:05.010005 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.009918 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/41e91d08-7c69-4959-ac36-fe5a10097787-kube-api-access-4wq7c\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:05.554811 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.554779 2565 generic.go:356] "Generic (PLEG): container finished" podID="41e91d08-7c69-4959-ac36-fe5a10097787" containerID="5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732" exitCode=0 Jul 08 18:25:05.555180 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.554830 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564949d647-sh6wf" Jul 08 18:25:05.555180 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.554860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564949d647-sh6wf" event={"ID":"41e91d08-7c69-4959-ac36-fe5a10097787","Type":"ContainerDied","Data":"5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732"} Jul 08 18:25:05.555180 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.554910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564949d647-sh6wf" event={"ID":"41e91d08-7c69-4959-ac36-fe5a10097787","Type":"ContainerDied","Data":"a7d3f4a1248c65eac0cf0bbc7fdbf566c276bbd26908de59620255cd79232f68"} Jul 08 18:25:05.555180 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.554930 2565 scope.go:117] "RemoveContainer" containerID="5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732" Jul 08 18:25:05.564042 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.564029 2565 scope.go:117] "RemoveContainer" containerID="5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732" Jul 08 18:25:05.564279 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:25:05.564262 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732\": container with ID starting with 5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732 not found: ID does not exist" containerID="5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732" Jul 08 18:25:05.564346 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.564285 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732"} err="failed to get container status \"5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732\": rpc error: code = NotFound desc = could not find container \"5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732\": container with ID starting with 5ba0ff926814c04b4fbb66acf5fa5e57f62784b31681b300f9eb859b4c6b1732 not found: ID does not exist" Jul 08 18:25:05.579088 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.579065 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564949d647-sh6wf"] Jul 08 18:25:05.583789 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.583771 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-564949d647-sh6wf"] Jul 08 18:25:05.932257 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:05.932187 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e91d08-7c69-4959-ac36-fe5a10097787" path="/var/lib/kubelet/pods/41e91d08-7c69-4959-ac36-fe5a10097787/volumes" Jul 08 18:25:27.196424 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.196389 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb"] Jul 08 18:25:27.196942 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.196845 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e91d08-7c69-4959-ac36-fe5a10097787" containerName="console" Jul 08 18:25:27.196942 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.196863 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e91d08-7c69-4959-ac36-fe5a10097787" containerName="console" Jul 08 18:25:27.197065 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.196955 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e91d08-7c69-4959-ac36-fe5a10097787" containerName="console" Jul 08 18:25:27.198897 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.198868 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.201308 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.201281 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 18:25:27.201400 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.201325 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 18:25:27.209864 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.209847 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xfzmc\"" Jul 08 18:25:27.215052 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.215032 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb"] Jul 08 18:25:27.358063 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.358038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnvl\" (UniqueName: \"kubernetes.io/projected/38071ddb-49d9-48de-b12e-6f5ead225e6b-kube-api-access-8cnvl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.358144 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.358075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.358144 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.358108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.459393 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.459335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.459393 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.459386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnvl\" (UniqueName: \"kubernetes.io/projected/38071ddb-49d9-48de-b12e-6f5ead225e6b-kube-api-access-8cnvl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.459508 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.459412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.459656 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.459639 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.459729 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.459690 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.467488 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.467471 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnvl\" (UniqueName: \"kubernetes.io/projected/38071ddb-49d9-48de-b12e-6f5ead225e6b-kube-api-access-8cnvl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.507732 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.507712 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:27.638249 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:27.638226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb"] Jul 08 18:25:27.640990 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:25:27.640964 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38071ddb_49d9_48de_b12e_6f5ead225e6b.slice/crio-5e99bfd0d7006d7134ca42f74a9173aadb96322f41514fcfe8e60d68505472e6 WatchSource:0}: Error finding container 5e99bfd0d7006d7134ca42f74a9173aadb96322f41514fcfe8e60d68505472e6: Status 404 returned error can't find the container with id 5e99bfd0d7006d7134ca42f74a9173aadb96322f41514fcfe8e60d68505472e6 Jul 08 18:25:28.648174 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:28.648131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" event={"ID":"38071ddb-49d9-48de-b12e-6f5ead225e6b","Type":"ContainerStarted","Data":"5e99bfd0d7006d7134ca42f74a9173aadb96322f41514fcfe8e60d68505472e6"} Jul 08 18:25:35.683366 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:35.683329 2565 generic.go:356] "Generic (PLEG): container finished" podID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerID="3e51538c1e3c1355a2b2f63b6a4ec61f11aa8c19b06f6042d5be89d8d3c2bad8" exitCode=0 Jul 08 18:25:35.683792 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:35.683419 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" event={"ID":"38071ddb-49d9-48de-b12e-6f5ead225e6b","Type":"ContainerDied","Data":"3e51538c1e3c1355a2b2f63b6a4ec61f11aa8c19b06f6042d5be89d8d3c2bad8"} Jul 08 18:25:37.693942 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:37.693906 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" event={"ID":"38071ddb-49d9-48de-b12e-6f5ead225e6b","Type":"ContainerStarted","Data":"1fb2a63b60d9016653331554dccc22c34c1b0d26b0eac1689ed200fd023f9738"} Jul 08 18:25:38.700184 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:38.700152 2565 generic.go:356] "Generic (PLEG): container finished" podID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerID="1fb2a63b60d9016653331554dccc22c34c1b0d26b0eac1689ed200fd023f9738" exitCode=0 Jul 08 18:25:38.700678 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:38.700194 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" event={"ID":"38071ddb-49d9-48de-b12e-6f5ead225e6b","Type":"ContainerDied","Data":"1fb2a63b60d9016653331554dccc22c34c1b0d26b0eac1689ed200fd023f9738"} Jul 08 18:25:47.747425 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:47.747395 2565 generic.go:356] "Generic (PLEG): container finished" podID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerID="d99b22f1635e3daec167d6e08fe44eb8e6b5b963d48a494ac64a8dd58715376d" exitCode=0 Jul 08 18:25:47.747807 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:47.747460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" event={"ID":"38071ddb-49d9-48de-b12e-6f5ead225e6b","Type":"ContainerDied","Data":"d99b22f1635e3daec167d6e08fe44eb8e6b5b963d48a494ac64a8dd58715376d"} Jul 08 18:25:48.916142 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:48.916116 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:49.023241 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.023184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-bundle\") pod \"38071ddb-49d9-48de-b12e-6f5ead225e6b\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " Jul 08 18:25:49.023241 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.023222 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cnvl\" (UniqueName: \"kubernetes.io/projected/38071ddb-49d9-48de-b12e-6f5ead225e6b-kube-api-access-8cnvl\") pod \"38071ddb-49d9-48de-b12e-6f5ead225e6b\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " Jul 08 18:25:49.023371 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.023248 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-util\") pod \"38071ddb-49d9-48de-b12e-6f5ead225e6b\" (UID: \"38071ddb-49d9-48de-b12e-6f5ead225e6b\") " Jul 08 18:25:49.023991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.023965 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-bundle" (OuterVolumeSpecName: "bundle") pod "38071ddb-49d9-48de-b12e-6f5ead225e6b" (UID: "38071ddb-49d9-48de-b12e-6f5ead225e6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:25:49.025430 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.025406 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38071ddb-49d9-48de-b12e-6f5ead225e6b-kube-api-access-8cnvl" (OuterVolumeSpecName: "kube-api-access-8cnvl") pod "38071ddb-49d9-48de-b12e-6f5ead225e6b" (UID: "38071ddb-49d9-48de-b12e-6f5ead225e6b"). InnerVolumeSpecName "kube-api-access-8cnvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:25:49.027247 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.027226 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-util" (OuterVolumeSpecName: "util") pod "38071ddb-49d9-48de-b12e-6f5ead225e6b" (UID: "38071ddb-49d9-48de-b12e-6f5ead225e6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:25:49.123984 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.123960 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-bundle\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:49.123984 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.123981 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnvl\" (UniqueName: \"kubernetes.io/projected/38071ddb-49d9-48de-b12e-6f5ead225e6b-kube-api-access-8cnvl\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:49.124107 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.123990 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38071ddb-49d9-48de-b12e-6f5ead225e6b-util\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:25:49.757388 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.757358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" event={"ID":"38071ddb-49d9-48de-b12e-6f5ead225e6b","Type":"ContainerDied","Data":"5e99bfd0d7006d7134ca42f74a9173aadb96322f41514fcfe8e60d68505472e6"} Jul 08 18:25:49.757388 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.757382 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pkbbb" Jul 08 18:25:49.757595 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:49.757403 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e99bfd0d7006d7134ca42f74a9173aadb96322f41514fcfe8e60d68505472e6" Jul 08 18:25:58.958429 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.958396 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4c72j"] Jul 08 18:25:58.958918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.958721 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerName="extract" Jul 08 18:25:58.958918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.958733 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerName="extract" Jul 08 18:25:58.958918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.958744 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerName="pull" Jul 08 18:25:58.958918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.958749 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerName="pull" Jul 08 18:25:58.958918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.958757 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerName="util" Jul 08 18:25:58.958918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.958762 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerName="util" Jul 08 18:25:58.958918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.958813 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38071ddb-49d9-48de-b12e-6f5ead225e6b" containerName="extract" Jul 08 18:25:58.960870 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.960853 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:25:58.963699 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.963655 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-68dnm\"" Jul 08 18:25:58.963838 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.963719 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 18:25:58.963838 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.963799 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 18:25:58.969677 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.969640 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4c72j"] Jul 08 18:25:58.997015 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.996986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mvk\" (UniqueName: \"kubernetes.io/projected/7bd660f5-f69e-4169-b34f-8b675e6ce670-kube-api-access-z6mvk\") pod \"cert-manager-webhook-6486ffd997-4c72j\" (UID: \"7bd660f5-f69e-4169-b34f-8b675e6ce670\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:25:58.997122 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:58.997108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bd660f5-f69e-4169-b34f-8b675e6ce670-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4c72j\" (UID: \"7bd660f5-f69e-4169-b34f-8b675e6ce670\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:25:59.097733 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.097711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bd660f5-f69e-4169-b34f-8b675e6ce670-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4c72j\" (UID: \"7bd660f5-f69e-4169-b34f-8b675e6ce670\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:25:59.097863 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.097777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mvk\" (UniqueName: \"kubernetes.io/projected/7bd660f5-f69e-4169-b34f-8b675e6ce670-kube-api-access-z6mvk\") pod \"cert-manager-webhook-6486ffd997-4c72j\" (UID: \"7bd660f5-f69e-4169-b34f-8b675e6ce670\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:25:59.106082 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.106062 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bd660f5-f69e-4169-b34f-8b675e6ce670-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4c72j\" (UID: \"7bd660f5-f69e-4169-b34f-8b675e6ce670\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:25:59.106219 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.106200 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mvk\" (UniqueName: \"kubernetes.io/projected/7bd660f5-f69e-4169-b34f-8b675e6ce670-kube-api-access-z6mvk\") pod \"cert-manager-webhook-6486ffd997-4c72j\" (UID: \"7bd660f5-f69e-4169-b34f-8b675e6ce670\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:25:59.285763 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.285738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:25:59.423817 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.423793 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4c72j"] Jul 08 18:25:59.425511 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:25:59.425479 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd660f5_f69e_4169_b34f_8b675e6ce670.slice/crio-8e8665cbb61862ef343ebe0bc0aadcc59f07a5d5d6e151022c1738dc44235ffd WatchSource:0}: Error finding container 8e8665cbb61862ef343ebe0bc0aadcc59f07a5d5d6e151022c1738dc44235ffd: Status 404 returned error can't find the container with id 8e8665cbb61862ef343ebe0bc0aadcc59f07a5d5d6e151022c1738dc44235ffd Jul 08 18:25:59.724626 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.724565 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb"] Jul 08 18:25:59.727503 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.727487 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.730018 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.729996 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 18:25:59.730132 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.730047 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 18:25:59.730703 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.730688 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xfzmc\"" Jul 08 18:25:59.738041 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.738019 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb"] Jul 08 18:25:59.798224 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.798203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" event={"ID":"7bd660f5-f69e-4169-b34f-8b675e6ce670","Type":"ContainerStarted","Data":"8e8665cbb61862ef343ebe0bc0aadcc59f07a5d5d6e151022c1738dc44235ffd"} Jul 08 18:25:59.803428 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.803410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.803530 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.803440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktsgq\" (UniqueName: \"kubernetes.io/projected/02b22121-1abb-4b8d-b946-fbfcc3253acf-kube-api-access-ktsgq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.803530 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.803471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.904390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.904367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.904482 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.904396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktsgq\" (UniqueName: \"kubernetes.io/projected/02b22121-1abb-4b8d-b946-fbfcc3253acf-kube-api-access-ktsgq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.904482 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.904424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.904763 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.904745 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.904837 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.904788 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:25:59.912792 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:25:59.912774 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktsgq\" (UniqueName: \"kubernetes.io/projected/02b22121-1abb-4b8d-b946-fbfcc3253acf-kube-api-access-ktsgq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:26:00.037102 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:00.037080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:26:00.192281 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:00.192228 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb"] Jul 08 18:26:00.194564 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:00.194542 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b22121_1abb_4b8d_b946_fbfcc3253acf.slice/crio-bb231ec42d95947cddcee37fbcfd8edc21d2f8d46c4515ae771750db0073df8d WatchSource:0}: Error finding container bb231ec42d95947cddcee37fbcfd8edc21d2f8d46c4515ae771750db0073df8d: Status 404 returned error can't find the container with id bb231ec42d95947cddcee37fbcfd8edc21d2f8d46c4515ae771750db0073df8d Jul 08 18:26:00.803804 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:00.803779 2565 generic.go:356] "Generic (PLEG): container finished" podID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerID="f866810bdb8d9b636ed5d8c65b34130c46e86cfa774ca9d5213bf05ba5641caf" exitCode=0 Jul 08 18:26:00.803953 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:00.803847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" event={"ID":"02b22121-1abb-4b8d-b946-fbfcc3253acf","Type":"ContainerDied","Data":"f866810bdb8d9b636ed5d8c65b34130c46e86cfa774ca9d5213bf05ba5641caf"} Jul 08 18:26:00.803953 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:00.803885 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" event={"ID":"02b22121-1abb-4b8d-b946-fbfcc3253acf","Type":"ContainerStarted","Data":"bb231ec42d95947cddcee37fbcfd8edc21d2f8d46c4515ae771750db0073df8d"} Jul 08 18:26:03.820319 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:03.820284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" event={"ID":"7bd660f5-f69e-4169-b34f-8b675e6ce670","Type":"ContainerStarted","Data":"de71aa701a48af9411bfa9ad9cdd247e0dc4acfdc35b5ba7e6becb31713b2438"} Jul 08 18:26:03.820646 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:03.820348 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:26:03.821888 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:03.821867 2565 generic.go:356] "Generic (PLEG): container finished" podID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerID="a7f380dc806f4ded71c68a5463005a104873a0ded0515eac454ed2357f9d6379" exitCode=0 Jul 08 18:26:03.821969 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:03.821938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" event={"ID":"02b22121-1abb-4b8d-b946-fbfcc3253acf","Type":"ContainerDied","Data":"a7f380dc806f4ded71c68a5463005a104873a0ded0515eac454ed2357f9d6379"} Jul 08 18:26:03.838711 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:03.838575 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" podStartSLOduration=1.586547093 podStartE2EDuration="5.838561895s" totalImagesPullingTime="4.252014802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:26:03.836580193 +0000 UTC m=+498.460167284" watchObservedRunningTime="2026-07-08 18:26:03.838561895 +0000 UTC m=+498.462148975" Jul 08 18:26:04.829052 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:04.829018 2565 generic.go:356] "Generic (PLEG): container finished" podID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerID="c70651cd98ee7e3778cc9fb52b7f55ab0f6d38d4f9256be832e21b1e787a36d2" exitCode=0 Jul 08 18:26:04.829458 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:04.829087 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" event={"ID":"02b22121-1abb-4b8d-b946-fbfcc3253acf","Type":"ContainerDied","Data":"c70651cd98ee7e3778cc9fb52b7f55ab0f6d38d4f9256be832e21b1e787a36d2"} Jul 08 18:26:05.748613 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.748583 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb"] Jul 08 18:26:05.752332 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.752316 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" Jul 08 18:26:05.754877 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.754857 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qncxl\"" Jul 08 18:26:05.760046 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.760026 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb"] Jul 08 18:26:05.848559 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.848532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2wh\" (UniqueName: \"kubernetes.io/projected/6ff54e64-0961-4e3c-9591-4e44c2ba0b63-kube-api-access-6d2wh\") pod \"cert-manager-cainjector-559cdd8c87-9cxmb\" (UID: \"6ff54e64-0961-4e3c-9591-4e44c2ba0b63\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" Jul 08 18:26:05.848926 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.848604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ff54e64-0961-4e3c-9591-4e44c2ba0b63-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9cxmb\" (UID: \"6ff54e64-0961-4e3c-9591-4e44c2ba0b63\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" Jul 08 18:26:05.949844 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.949822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2wh\" (UniqueName: \"kubernetes.io/projected/6ff54e64-0961-4e3c-9591-4e44c2ba0b63-kube-api-access-6d2wh\") pod \"cert-manager-cainjector-559cdd8c87-9cxmb\" (UID: \"6ff54e64-0961-4e3c-9591-4e44c2ba0b63\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" Jul 08 18:26:05.949966 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.949860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ff54e64-0961-4e3c-9591-4e44c2ba0b63-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9cxmb\" (UID: \"6ff54e64-0961-4e3c-9591-4e44c2ba0b63\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" Jul 08 18:26:05.958542 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.958512 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ff54e64-0961-4e3c-9591-4e44c2ba0b63-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9cxmb\" (UID: \"6ff54e64-0961-4e3c-9591-4e44c2ba0b63\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" Jul 08 18:26:05.958767 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.958749 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2wh\" (UniqueName: \"kubernetes.io/projected/6ff54e64-0961-4e3c-9591-4e44c2ba0b63-kube-api-access-6d2wh\") pod \"cert-manager-cainjector-559cdd8c87-9cxmb\" (UID: \"6ff54e64-0961-4e3c-9591-4e44c2ba0b63\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" Jul 08 18:26:05.962014 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:05.961995 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:26:06.050574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.050522 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktsgq\" (UniqueName: \"kubernetes.io/projected/02b22121-1abb-4b8d-b946-fbfcc3253acf-kube-api-access-ktsgq\") pod \"02b22121-1abb-4b8d-b946-fbfcc3253acf\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " Jul 08 18:26:06.050691 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.050591 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-bundle\") pod \"02b22121-1abb-4b8d-b946-fbfcc3253acf\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " Jul 08 18:26:06.050691 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.050618 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-util\") pod \"02b22121-1abb-4b8d-b946-fbfcc3253acf\" (UID: \"02b22121-1abb-4b8d-b946-fbfcc3253acf\") " Jul 08 18:26:06.051007 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.050986 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-bundle" (OuterVolumeSpecName: "bundle") pod "02b22121-1abb-4b8d-b946-fbfcc3253acf" (UID: "02b22121-1abb-4b8d-b946-fbfcc3253acf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:26:06.052454 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.052436 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b22121-1abb-4b8d-b946-fbfcc3253acf-kube-api-access-ktsgq" (OuterVolumeSpecName: "kube-api-access-ktsgq") pod "02b22121-1abb-4b8d-b946-fbfcc3253acf" (UID: "02b22121-1abb-4b8d-b946-fbfcc3253acf"). InnerVolumeSpecName "kube-api-access-ktsgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:26:06.054597 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.054570 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-util" (OuterVolumeSpecName: "util") pod "02b22121-1abb-4b8d-b946-fbfcc3253acf" (UID: "02b22121-1abb-4b8d-b946-fbfcc3253acf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:26:06.061565 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.061548 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" Jul 08 18:26:06.151452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.151422 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-bundle\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:26:06.151452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.151455 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02b22121-1abb-4b8d-b946-fbfcc3253acf-util\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:26:06.151607 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.151470 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktsgq\" (UniqueName: \"kubernetes.io/projected/02b22121-1abb-4b8d-b946-fbfcc3253acf-kube-api-access-ktsgq\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:26:06.191115 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.191090 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb"] Jul 08 18:26:06.193563 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:06.193538 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ff54e64_0961_4e3c_9591_4e44c2ba0b63.slice/crio-8eeeab74798f7f0b99176cf0401ff1e6a76cff78b5c2764cd7509d77c4ce21b7 WatchSource:0}: Error finding container 8eeeab74798f7f0b99176cf0401ff1e6a76cff78b5c2764cd7509d77c4ce21b7: Status 404 returned error can't find the container with id 8eeeab74798f7f0b99176cf0401ff1e6a76cff78b5c2764cd7509d77c4ce21b7 Jul 08 18:26:06.840348 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.840317 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" event={"ID":"02b22121-1abb-4b8d-b946-fbfcc3253acf","Type":"ContainerDied","Data":"bb231ec42d95947cddcee37fbcfd8edc21d2f8d46c4515ae771750db0073df8d"} Jul 08 18:26:06.840538 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.840372 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb231ec42d95947cddcee37fbcfd8edc21d2f8d46c4515ae771750db0073df8d" Jul 08 18:26:06.840538 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.840333 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh9zb" Jul 08 18:26:06.842016 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.841988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" event={"ID":"6ff54e64-0961-4e3c-9591-4e44c2ba0b63","Type":"ContainerStarted","Data":"6c1254492554643aaa2dbb16a5c73a53c937658a0d7be2a62a7cbd709f0657bd"} Jul 08 18:26:06.842166 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.842026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" event={"ID":"6ff54e64-0961-4e3c-9591-4e44c2ba0b63","Type":"ContainerStarted","Data":"8eeeab74798f7f0b99176cf0401ff1e6a76cff78b5c2764cd7509d77c4ce21b7"} Jul 08 18:26:06.858786 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:06.858749 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9cxmb" podStartSLOduration=1.858733996 podStartE2EDuration="1.858733996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:26:06.85743154 +0000 UTC m=+501.481018623" watchObservedRunningTime="2026-07-08 18:26:06.858733996 +0000 UTC m=+501.482321078" Jul 08 18:26:09.832059 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:09.832021 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4c72j" Jul 08 18:26:17.481057 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.481025 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj"] Jul 08 18:26:17.481519 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.481333 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerName="util" Jul 08 18:26:17.481519 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.481345 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerName="util" Jul 08 18:26:17.481519 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.481367 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerName="pull" Jul 08 18:26:17.481519 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.481373 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerName="pull" Jul 08 18:26:17.481519 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.481383 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerName="extract" Jul 08 18:26:17.481519 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.481388 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerName="extract" Jul 08 18:26:17.481519 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.481438 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b22121-1abb-4b8d-b946-fbfcc3253acf" containerName="extract" Jul 08 18:26:17.485825 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.485806 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" Jul 08 18:26:17.492556 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.492526 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:26:17.493279 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.493263 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 18:26:17.493341 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.493288 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-8vfzf\"" Jul 08 18:26:17.498808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.498783 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj"] Jul 08 18:26:17.532414 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.532389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c9b2e84-9097-497a-b057-94e0d7cbb449-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9xhgj\" (UID: \"0c9b2e84-9097-497a-b057-94e0d7cbb449\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" Jul 08 18:26:17.532524 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.532419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vs9\" (UniqueName: \"kubernetes.io/projected/0c9b2e84-9097-497a-b057-94e0d7cbb449-kube-api-access-j9vs9\") pod \"openshift-lws-operator-fd8ccff4c-9xhgj\" (UID: \"0c9b2e84-9097-497a-b057-94e0d7cbb449\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" Jul 08 18:26:17.632840 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.632811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c9b2e84-9097-497a-b057-94e0d7cbb449-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9xhgj\" (UID: \"0c9b2e84-9097-497a-b057-94e0d7cbb449\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" Jul 08 18:26:17.632957 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.632851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vs9\" (UniqueName: \"kubernetes.io/projected/0c9b2e84-9097-497a-b057-94e0d7cbb449-kube-api-access-j9vs9\") pod \"openshift-lws-operator-fd8ccff4c-9xhgj\" (UID: \"0c9b2e84-9097-497a-b057-94e0d7cbb449\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" Jul 08 18:26:17.633166 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.633147 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c9b2e84-9097-497a-b057-94e0d7cbb449-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9xhgj\" (UID: \"0c9b2e84-9097-497a-b057-94e0d7cbb449\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" Jul 08 18:26:17.641472 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.641455 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vs9\" (UniqueName: \"kubernetes.io/projected/0c9b2e84-9097-497a-b057-94e0d7cbb449-kube-api-access-j9vs9\") pod \"openshift-lws-operator-fd8ccff4c-9xhgj\" (UID: \"0c9b2e84-9097-497a-b057-94e0d7cbb449\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" Jul 08 18:26:17.801802 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.801782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" Jul 08 18:26:17.930984 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:17.930959 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj"] Jul 08 18:26:17.932306 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:17.932283 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c9b2e84_9097_497a_b057_94e0d7cbb449.slice/crio-f058c56cc252fa706fcf8848af5f96dc9190b1b76a3ad316a2ee0e56519f8832 WatchSource:0}: Error finding container f058c56cc252fa706fcf8848af5f96dc9190b1b76a3ad316a2ee0e56519f8832: Status 404 returned error can't find the container with id f058c56cc252fa706fcf8848af5f96dc9190b1b76a3ad316a2ee0e56519f8832 Jul 08 18:26:18.899094 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:18.899047 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" event={"ID":"0c9b2e84-9097-497a-b057-94e0d7cbb449","Type":"ContainerStarted","Data":"f058c56cc252fa706fcf8848af5f96dc9190b1b76a3ad316a2ee0e56519f8832"} Jul 08 18:26:19.905305 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:19.905227 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" event={"ID":"0c9b2e84-9097-497a-b057-94e0d7cbb449","Type":"ContainerStarted","Data":"44fc1705d311c3ee8afe744f926640407f9d236bc64bcff596eeaefd1f6d3471"} Jul 08 18:26:19.922417 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:19.922373 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xhgj" podStartSLOduration=1.282495494 podStartE2EDuration="2.922359731s" totalImagesPullingTime="1.639864237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:26:19.921517453 +0000 UTC m=+514.545104532" watchObservedRunningTime="2026-07-08 18:26:19.922359731 +0000 UTC m=+514.545946811" Jul 08 18:26:22.518606 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.518568 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg"] Jul 08 18:26:22.521170 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.521154 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.523710 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.523686 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 18:26:22.524781 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.524760 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 18:26:22.524887 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.524829 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xfzmc\"" Jul 08 18:26:22.531832 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.531804 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg"] Jul 08 18:26:22.573516 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.573487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.573684 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.573531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhwt\" (UniqueName: \"kubernetes.io/projected/681df04b-6752-4330-930a-1a9dda031cd6-kube-api-access-lfhwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.573684 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.573598 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.674731 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.674703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.674852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.674745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhwt\" (UniqueName: \"kubernetes.io/projected/681df04b-6752-4330-930a-1a9dda031cd6-kube-api-access-lfhwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.674852 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.674794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.675089 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.675069 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.675154 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.675100 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.682933 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.682900 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhwt\" (UniqueName: \"kubernetes.io/projected/681df04b-6752-4330-930a-1a9dda031cd6-kube-api-access-lfhwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.830863 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.830793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:22.956375 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:22.956348 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg"] Jul 08 18:26:22.958584 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:22.958563 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681df04b_6752_4330_930a_1a9dda031cd6.slice/crio-7db9e3449fd9ed1191f272037f650216a3ed8663e288091311e1dec25520db2f WatchSource:0}: Error finding container 7db9e3449fd9ed1191f272037f650216a3ed8663e288091311e1dec25520db2f: Status 404 returned error can't find the container with id 7db9e3449fd9ed1191f272037f650216a3ed8663e288091311e1dec25520db2f Jul 08 18:26:23.926467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:23.926436 2565 generic.go:356] "Generic (PLEG): container finished" podID="681df04b-6752-4330-930a-1a9dda031cd6" containerID="3d03897e20ad67407583bf7e08eb8e602aa38d65e41c7840e2b2e54d7ce24e27" exitCode=0 Jul 08 18:26:23.927302 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:23.927277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" event={"ID":"681df04b-6752-4330-930a-1a9dda031cd6","Type":"ContainerDied","Data":"3d03897e20ad67407583bf7e08eb8e602aa38d65e41c7840e2b2e54d7ce24e27"} Jul 08 18:26:23.927365 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:23.927324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" event={"ID":"681df04b-6752-4330-930a-1a9dda031cd6","Type":"ContainerStarted","Data":"7db9e3449fd9ed1191f272037f650216a3ed8663e288091311e1dec25520db2f"} Jul 08 18:26:24.932913 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:24.932853 2565 generic.go:356] "Generic (PLEG): container finished" podID="681df04b-6752-4330-930a-1a9dda031cd6" containerID="1623367f7392b280c939f2caf2d1777fea956f54aca370b21c9b4b2ecb0bbee2" exitCode=0 Jul 08 18:26:24.933324 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:24.932904 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" event={"ID":"681df04b-6752-4330-930a-1a9dda031cd6","Type":"ContainerDied","Data":"1623367f7392b280c939f2caf2d1777fea956f54aca370b21c9b4b2ecb0bbee2"} Jul 08 18:26:25.940494 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:25.940463 2565 generic.go:356] "Generic (PLEG): container finished" podID="681df04b-6752-4330-930a-1a9dda031cd6" containerID="f5bfa3ee58c39c4b392cd2fd8033fe701834dd51360fa0503e1bb7f1e5fb2b18" exitCode=0 Jul 08 18:26:25.940865 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:25.940541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" event={"ID":"681df04b-6752-4330-930a-1a9dda031cd6","Type":"ContainerDied","Data":"f5bfa3ee58c39c4b392cd2fd8033fe701834dd51360fa0503e1bb7f1e5fb2b18"} Jul 08 18:26:27.073655 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.073634 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:27.203638 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.203583 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-util\") pod \"681df04b-6752-4330-930a-1a9dda031cd6\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " Jul 08 18:26:27.203638 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.203626 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-bundle\") pod \"681df04b-6752-4330-930a-1a9dda031cd6\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " Jul 08 18:26:27.203842 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.203701 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfhwt\" (UniqueName: \"kubernetes.io/projected/681df04b-6752-4330-930a-1a9dda031cd6-kube-api-access-lfhwt\") pod \"681df04b-6752-4330-930a-1a9dda031cd6\" (UID: \"681df04b-6752-4330-930a-1a9dda031cd6\") " Jul 08 18:26:27.204371 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.204343 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-bundle" (OuterVolumeSpecName: "bundle") pod "681df04b-6752-4330-930a-1a9dda031cd6" (UID: "681df04b-6752-4330-930a-1a9dda031cd6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:26:27.205708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.205686 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681df04b-6752-4330-930a-1a9dda031cd6-kube-api-access-lfhwt" (OuterVolumeSpecName: "kube-api-access-lfhwt") pod "681df04b-6752-4330-930a-1a9dda031cd6" (UID: "681df04b-6752-4330-930a-1a9dda031cd6"). InnerVolumeSpecName "kube-api-access-lfhwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:26:27.209142 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.209121 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-util" (OuterVolumeSpecName: "util") pod "681df04b-6752-4330-930a-1a9dda031cd6" (UID: "681df04b-6752-4330-930a-1a9dda031cd6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:26:27.304570 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.304550 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhwt\" (UniqueName: \"kubernetes.io/projected/681df04b-6752-4330-930a-1a9dda031cd6-kube-api-access-lfhwt\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:26:27.304570 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.304571 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-util\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:26:27.304716 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.304580 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/681df04b-6752-4330-930a-1a9dda031cd6-bundle\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:26:27.951038 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.951011 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" Jul 08 18:26:27.951149 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.951012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxncjg" event={"ID":"681df04b-6752-4330-930a-1a9dda031cd6","Type":"ContainerDied","Data":"7db9e3449fd9ed1191f272037f650216a3ed8663e288091311e1dec25520db2f"} Jul 08 18:26:27.951149 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:27.951127 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7db9e3449fd9ed1191f272037f650216a3ed8663e288091311e1dec25520db2f" Jul 08 18:26:36.310241 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.310205 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb"] Jul 08 18:26:36.310611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.310529 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681df04b-6752-4330-930a-1a9dda031cd6" containerName="extract" Jul 08 18:26:36.310611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.310542 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="681df04b-6752-4330-930a-1a9dda031cd6" containerName="extract" Jul 08 18:26:36.310611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.310564 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681df04b-6752-4330-930a-1a9dda031cd6" containerName="pull" Jul 08 18:26:36.310611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.310585 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="681df04b-6752-4330-930a-1a9dda031cd6" containerName="pull" Jul 08 18:26:36.310611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.310594 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681df04b-6752-4330-930a-1a9dda031cd6" containerName="util" Jul 08 18:26:36.310611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.310600 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="681df04b-6752-4330-930a-1a9dda031cd6" containerName="util" Jul 08 18:26:36.310816 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.310674 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="681df04b-6752-4330-930a-1a9dda031cd6" containerName="extract" Jul 08 18:26:36.313025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.313008 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.316085 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.316051 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 18:26:36.316509 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.316493 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 18:26:36.316620 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.316525 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q6glk\"" Jul 08 18:26:36.316620 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.316539 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 18:26:36.316922 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.316908 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 18:26:36.329132 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.329105 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb"] Jul 08 18:26:36.474623 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.474600 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee24932a-c92e-4f69-83de-8de67c558069-manifests\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.474751 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.474630 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee24932a-c92e-4f69-83de-8de67c558069-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.474751 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.474648 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdf9s\" (UniqueName: \"kubernetes.io/projected/ee24932a-c92e-4f69-83de-8de67c558069-kube-api-access-kdf9s\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.474751 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.474686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee24932a-c92e-4f69-83de-8de67c558069-webhook-cert\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.474751 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.474744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee24932a-c92e-4f69-83de-8de67c558069-tmp\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.575574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.575504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee24932a-c92e-4f69-83de-8de67c558069-tmp\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.575574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.575571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee24932a-c92e-4f69-83de-8de67c558069-manifests\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.575743 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.575599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee24932a-c92e-4f69-83de-8de67c558069-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.575743 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.575707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdf9s\" (UniqueName: \"kubernetes.io/projected/ee24932a-c92e-4f69-83de-8de67c558069-kube-api-access-kdf9s\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.575743 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.575741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee24932a-c92e-4f69-83de-8de67c558069-webhook-cert\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.575908 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.575886 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee24932a-c92e-4f69-83de-8de67c558069-manifests\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.578040 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.578013 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee24932a-c92e-4f69-83de-8de67c558069-tmp\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.578194 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.578177 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee24932a-c92e-4f69-83de-8de67c558069-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.578236 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.578211 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee24932a-c92e-4f69-83de-8de67c558069-webhook-cert\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.587997 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.587977 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdf9s\" (UniqueName: \"kubernetes.io/projected/ee24932a-c92e-4f69-83de-8de67c558069-kube-api-access-kdf9s\") pod \"opendatahub-operator-controller-manager-578597fb9c-87mtb\" (UID: \"ee24932a-c92e-4f69-83de-8de67c558069\") " pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.622883 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.622861 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:36.770131 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.770102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb"] Jul 08 18:26:36.771570 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:36.771543 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee24932a_c92e_4f69_83de_8de67c558069.slice/crio-c83c73540d0f98c6776b33391a30c6cecbdecb66082b82b8bbd48cdb6561d141 WatchSource:0}: Error finding container c83c73540d0f98c6776b33391a30c6cecbdecb66082b82b8bbd48cdb6561d141: Status 404 returned error can't find the container with id c83c73540d0f98c6776b33391a30c6cecbdecb66082b82b8bbd48cdb6561d141 Jul 08 18:26:36.994010 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:36.993982 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" event={"ID":"ee24932a-c92e-4f69-83de-8de67c558069","Type":"ContainerStarted","Data":"c83c73540d0f98c6776b33391a30c6cecbdecb66082b82b8bbd48cdb6561d141"} Jul 08 18:26:40.011870 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:40.011842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" event={"ID":"ee24932a-c92e-4f69-83de-8de67c558069","Type":"ContainerStarted","Data":"aa29d798b99bc73987d8b8ecb400cdb76d188c1ac09de5820e3a247b102c5a81"} Jul 08 18:26:41.018047 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:41.018010 2565 generic.go:356] "Generic (PLEG): container finished" podID="ee24932a-c92e-4f69-83de-8de67c558069" containerID="aa29d798b99bc73987d8b8ecb400cdb76d188c1ac09de5820e3a247b102c5a81" exitCode=0 Jul 08 18:26:41.018394 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:41.018069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" event={"ID":"ee24932a-c92e-4f69-83de-8de67c558069","Type":"ContainerDied","Data":"aa29d798b99bc73987d8b8ecb400cdb76d188c1ac09de5820e3a247b102c5a81"} Jul 08 18:26:42.025542 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:42.025511 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" event={"ID":"ee24932a-c92e-4f69-83de-8de67c558069","Type":"ContainerStarted","Data":"15cf96f87321503e92c468c9e8b3f252722dd2e4089c9965ac0965ad6f2d37c9"} Jul 08 18:26:42.025925 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:42.025602 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:42.051435 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:42.051382 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" podStartSLOduration=1.816125729 podStartE2EDuration="6.051363328s" totalImagesPullingTime="3.239937839s" totalInitContainerRuntime="995.29976ms" isStatefulPod=false podCreationTimestamp="2026-07-08 18:26:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:26:42.0486025 +0000 UTC m=+536.672189630" watchObservedRunningTime="2026-07-08 18:26:42.051363328 +0000 UTC m=+536.674950410" Jul 08 18:26:53.033984 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:53.033949 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-578597fb9c-87mtb" Jul 08 18:26:55.782083 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.782049 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb94956b8-s895h"] Jul 08 18:26:55.792955 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.792914 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:55.797230 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.797200 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 18:26:55.798270 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.798249 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 18:26:55.798437 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.798326 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 18:26:55.798437 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.798348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zk85b\"" Jul 08 18:26:55.798624 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.798420 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 18:26:55.801065 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.801048 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb94956b8-s895h"] Jul 08 18:26:55.920892 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.920863 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-tlmk4"] Jul 08 18:26:55.921155 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.921133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvtl\" (UniqueName: \"kubernetes.io/projected/c2957457-c64b-4f9a-b886-7b08c4fa7902-kube-api-access-svvtl\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:55.921207 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.921169 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2957457-c64b-4f9a-b886-7b08c4fa7902-tls-certs\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:55.921207 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.921195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2957457-c64b-4f9a-b886-7b08c4fa7902-tmp\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:55.923957 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.923943 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:55.927054 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.927014 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 18:26:55.927221 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.927030 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-j8qv7\"" Jul 08 18:26:55.936345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:55.936323 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-tlmk4"] Jul 08 18:26:56.022401 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.022359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dd4082d-29d8-495e-8774-fe298a1c6a4b-cert\") pod \"odh-model-controller-7998c67dc4-tlmk4\" (UID: \"2dd4082d-29d8-495e-8774-fe298a1c6a4b\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:56.022574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.022403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvtl\" (UniqueName: \"kubernetes.io/projected/c2957457-c64b-4f9a-b886-7b08c4fa7902-kube-api-access-svvtl\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:56.022574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.022443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2957457-c64b-4f9a-b886-7b08c4fa7902-tls-certs\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:56.022574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.022483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2957457-c64b-4f9a-b886-7b08c4fa7902-tmp\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:56.022574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.022524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwm2\" (UniqueName: \"kubernetes.io/projected/2dd4082d-29d8-495e-8774-fe298a1c6a4b-kube-api-access-wzwm2\") pod \"odh-model-controller-7998c67dc4-tlmk4\" (UID: \"2dd4082d-29d8-495e-8774-fe298a1c6a4b\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:56.025280 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.025254 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2957457-c64b-4f9a-b886-7b08c4fa7902-tmp\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:56.025548 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.025525 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2957457-c64b-4f9a-b886-7b08c4fa7902-tls-certs\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:56.038786 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.038725 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvtl\" (UniqueName: \"kubernetes.io/projected/c2957457-c64b-4f9a-b886-7b08c4fa7902-kube-api-access-svvtl\") pod \"kube-auth-proxy-5fb94956b8-s895h\" (UID: \"c2957457-c64b-4f9a-b886-7b08c4fa7902\") " pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:56.103206 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.103182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" Jul 08 18:26:56.123808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.123775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dd4082d-29d8-495e-8774-fe298a1c6a4b-cert\") pod \"odh-model-controller-7998c67dc4-tlmk4\" (UID: \"2dd4082d-29d8-495e-8774-fe298a1c6a4b\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:56.123935 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.123874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwm2\" (UniqueName: \"kubernetes.io/projected/2dd4082d-29d8-495e-8774-fe298a1c6a4b-kube-api-access-wzwm2\") pod \"odh-model-controller-7998c67dc4-tlmk4\" (UID: \"2dd4082d-29d8-495e-8774-fe298a1c6a4b\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:56.123935 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:26:56.123925 2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 08 18:26:56.124046 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:26:56.123991 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd4082d-29d8-495e-8774-fe298a1c6a4b-cert podName:2dd4082d-29d8-495e-8774-fe298a1c6a4b nodeName:}" failed. No retries permitted until 2026-07-08 18:26:56.623972907 +0000 UTC m=+551.247559986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2dd4082d-29d8-495e-8774-fe298a1c6a4b-cert") pod "odh-model-controller-7998c67dc4-tlmk4" (UID: "2dd4082d-29d8-495e-8774-fe298a1c6a4b") : secret "odh-model-controller-webhook-cert" not found Jul 08 18:26:56.144545 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.144508 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwm2\" (UniqueName: \"kubernetes.io/projected/2dd4082d-29d8-495e-8774-fe298a1c6a4b-kube-api-access-wzwm2\") pod \"odh-model-controller-7998c67dc4-tlmk4\" (UID: \"2dd4082d-29d8-495e-8774-fe298a1c6a4b\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:56.253370 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.253336 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb94956b8-s895h"] Jul 08 18:26:56.257141 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:56.257109 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2957457_c64b_4f9a_b886_7b08c4fa7902.slice/crio-635b1b437ca405638cde5ca0a168a50f13f77a676a04ae06bb3ece2693305549 WatchSource:0}: Error finding container 635b1b437ca405638cde5ca0a168a50f13f77a676a04ae06bb3ece2693305549: Status 404 returned error can't find the container with id 635b1b437ca405638cde5ca0a168a50f13f77a676a04ae06bb3ece2693305549 Jul 08 18:26:56.628456 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.628420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dd4082d-29d8-495e-8774-fe298a1c6a4b-cert\") pod \"odh-model-controller-7998c67dc4-tlmk4\" (UID: \"2dd4082d-29d8-495e-8774-fe298a1c6a4b\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:56.630971 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.630938 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dd4082d-29d8-495e-8774-fe298a1c6a4b-cert\") pod \"odh-model-controller-7998c67dc4-tlmk4\" (UID: \"2dd4082d-29d8-495e-8774-fe298a1c6a4b\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:56.701150 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.701117 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x59v7"] Jul 08 18:26:56.703935 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.703917 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:56.707482 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.707464 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 18:26:56.707586 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.707548 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hn287\"" Jul 08 18:26:56.724755 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.724720 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x59v7"] Jul 08 18:26:56.830484 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.830450 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5w7\" (UniqueName: \"kubernetes.io/projected/9d992c01-aead-4782-8b0e-df526cf1453f-kube-api-access-dm5w7\") pod \"model-serving-api-54d89bd5fc-x59v7\" (UID: \"9d992c01-aead-4782-8b0e-df526cf1453f\") " pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:56.830890 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.830570 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d992c01-aead-4782-8b0e-df526cf1453f-tls-certs\") pod \"model-serving-api-54d89bd5fc-x59v7\" (UID: \"9d992c01-aead-4782-8b0e-df526cf1453f\") " pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:56.836331 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.836310 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:26:56.932574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.931628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d992c01-aead-4782-8b0e-df526cf1453f-tls-certs\") pod \"model-serving-api-54d89bd5fc-x59v7\" (UID: \"9d992c01-aead-4782-8b0e-df526cf1453f\") " pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:56.932574 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.931732 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5w7\" (UniqueName: \"kubernetes.io/projected/9d992c01-aead-4782-8b0e-df526cf1453f-kube-api-access-dm5w7\") pod \"model-serving-api-54d89bd5fc-x59v7\" (UID: \"9d992c01-aead-4782-8b0e-df526cf1453f\") " pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:56.932574 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:26:56.932151 2565 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 08 18:26:56.932574 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:26:56.932216 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d992c01-aead-4782-8b0e-df526cf1453f-tls-certs podName:9d992c01-aead-4782-8b0e-df526cf1453f nodeName:}" failed. No retries permitted until 2026-07-08 18:26:57.432196578 +0000 UTC m=+552.055783639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9d992c01-aead-4782-8b0e-df526cf1453f-tls-certs") pod "model-serving-api-54d89bd5fc-x59v7" (UID: "9d992c01-aead-4782-8b0e-df526cf1453f") : secret "model-serving-api-tls" not found Jul 08 18:26:56.949054 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.948978 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5w7\" (UniqueName: \"kubernetes.io/projected/9d992c01-aead-4782-8b0e-df526cf1453f-kube-api-access-dm5w7\") pod \"model-serving-api-54d89bd5fc-x59v7\" (UID: \"9d992c01-aead-4782-8b0e-df526cf1453f\") " pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:56.987406 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:56.987382 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-tlmk4"] Jul 08 18:26:56.989099 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:56.989075 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd4082d_29d8_495e_8774_fe298a1c6a4b.slice/crio-1324e4c366d2c29a3c1c09d9b1c4bd27091eea21f573f2260b1f11108d683aaf WatchSource:0}: Error finding container 1324e4c366d2c29a3c1c09d9b1c4bd27091eea21f573f2260b1f11108d683aaf: Status 404 returned error can't find the container with id 1324e4c366d2c29a3c1c09d9b1c4bd27091eea21f573f2260b1f11108d683aaf Jul 08 18:26:57.076248 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.076219 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sx74s"] Jul 08 18:26:57.079228 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.079208 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.081968 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.081944 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 18:26:57.082282 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.082259 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pkdqr\"" Jul 08 18:26:57.082430 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.082293 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 18:26:57.082528 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.082324 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 18:26:57.082579 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.082340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 18:26:57.093655 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.093637 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sx74s"] Jul 08 18:26:57.098473 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.098449 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" event={"ID":"2dd4082d-29d8-495e-8774-fe298a1c6a4b","Type":"ContainerStarted","Data":"1324e4c366d2c29a3c1c09d9b1c4bd27091eea21f573f2260b1f11108d683aaf"} Jul 08 18:26:57.099588 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.099556 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" event={"ID":"c2957457-c64b-4f9a-b886-7b08c4fa7902","Type":"ContainerStarted","Data":"635b1b437ca405638cde5ca0a168a50f13f77a676a04ae06bb3ece2693305549"} Jul 08 18:26:57.234285 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.234193 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-local-certs\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.234452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.234316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.234452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.234358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.234452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.234418 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.234452 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.234449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-token\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.234671 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.234483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-kube-api-access-9b9ns\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.234671 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.234549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-cacerts\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.335738 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.335706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-local-certs\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.335904 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.335790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.335904 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.335818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.335904 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.335873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.335904 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.335899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-token\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.336080 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.335935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-kube-api-access-9b9ns\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.336080 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.335998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-cacerts\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.336880 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.336828 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.340007 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.339006 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-local-certs\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.340007 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.339057 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.340007 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.339655 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-cacerts\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.341433 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.341388 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.348551 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.348511 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-token\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.350190 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.350169 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-kube-api-access-9b9ns\") pod \"istiod-openshift-gateway-7574459656-sx74s\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.388774 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.388751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:26:57.436524 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.436489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d992c01-aead-4782-8b0e-df526cf1453f-tls-certs\") pod \"model-serving-api-54d89bd5fc-x59v7\" (UID: \"9d992c01-aead-4782-8b0e-df526cf1453f\") " pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:57.440490 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.440460 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d992c01-aead-4782-8b0e-df526cf1453f-tls-certs\") pod \"model-serving-api-54d89bd5fc-x59v7\" (UID: \"9d992c01-aead-4782-8b0e-df526cf1453f\") " pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:57.547138 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.546771 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sx74s"] Jul 08 18:26:57.549441 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:57.549411 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa4b05e_bfb6_4ce1_b463_012e137b1d5e.slice/crio-1b5dc3b1523a048256338fa6d33186040ecbfd482033235eddf749a5706e4cb9 WatchSource:0}: Error finding container 1b5dc3b1523a048256338fa6d33186040ecbfd482033235eddf749a5706e4cb9: Status 404 returned error can't find the container with id 1b5dc3b1523a048256338fa6d33186040ecbfd482033235eddf749a5706e4cb9 Jul 08 18:26:57.616636 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.616598 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:26:57.761704 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:57.761680 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x59v7"] Jul 08 18:26:57.764127 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:26:57.764104 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d992c01_aead_4782_8b0e_df526cf1453f.slice/crio-64e067fd1887fb28a1a7a31fad4cda61b5cf361ead0e9536f6a6c960ea4cd2ed WatchSource:0}: Error finding container 64e067fd1887fb28a1a7a31fad4cda61b5cf361ead0e9536f6a6c960ea4cd2ed: Status 404 returned error can't find the container with id 64e067fd1887fb28a1a7a31fad4cda61b5cf361ead0e9536f6a6c960ea4cd2ed Jul 08 18:26:58.106946 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:58.106868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" event={"ID":"9d992c01-aead-4782-8b0e-df526cf1453f","Type":"ContainerStarted","Data":"64e067fd1887fb28a1a7a31fad4cda61b5cf361ead0e9536f6a6c960ea4cd2ed"} Jul 08 18:26:58.108304 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:26:58.108275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" event={"ID":"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e","Type":"ContainerStarted","Data":"1b5dc3b1523a048256338fa6d33186040ecbfd482033235eddf749a5706e4cb9"} Jul 08 18:27:02.484235 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:02.484152 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:27:02.484235 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:02.484213 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:27:03.148778 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.148420 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" event={"ID":"2dd4082d-29d8-495e-8774-fe298a1c6a4b","Type":"ContainerStarted","Data":"4ce75eec83ae65875e180f32f47f8eb1a12fd576252fabd9b51cfabf4c0c7e7e"} Jul 08 18:27:03.148778 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.148686 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:27:03.150841 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.150802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" event={"ID":"c2957457-c64b-4f9a-b886-7b08c4fa7902","Type":"ContainerStarted","Data":"3e4fbd2205fcf236b6fd0f9aa24e5db171032da14b8d2e02f459dac8b22677d4"} Jul 08 18:27:03.153629 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.153600 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" event={"ID":"9d992c01-aead-4782-8b0e-df526cf1453f","Type":"ContainerStarted","Data":"0f628ae7ea3e4d35af78423696949dedc1c5473ec1f88bcf4242b0aca1df84fe"} Jul 08 18:27:03.154097 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.154077 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:27:03.156173 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.156143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" event={"ID":"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e","Type":"ContainerStarted","Data":"2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912"} Jul 08 18:27:03.156968 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.156857 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:27:03.159715 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.159591 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-sx74s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 18:27:03.159715 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.159639 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" podUID="7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 18:27:03.173278 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.171309 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" podStartSLOduration=2.672467157 podStartE2EDuration="8.171289163s" totalImagesPullingTime="5.498822006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:27:03.169731059 +0000 UTC m=+557.793318138" watchObservedRunningTime="2026-07-08 18:27:03.171289163 +0000 UTC m=+557.794876244" Jul 08 18:27:03.204975 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.204928 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fb94956b8-s895h" podStartSLOduration=1.98053538 podStartE2EDuration="8.204913226s" totalImagesPullingTime="6.224377846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:27:03.20272167 +0000 UTC m=+557.826308751" watchObservedRunningTime="2026-07-08 18:27:03.204913226 +0000 UTC m=+557.828500305" Jul 08 18:27:03.249108 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.249048 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" podStartSLOduration=1.317708932 podStartE2EDuration="6.249029439s" totalImagesPullingTime="4.931320507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:27:03.245876378 +0000 UTC m=+557.869463475" watchObservedRunningTime="2026-07-08 18:27:03.249029439 +0000 UTC m=+557.872616518" Jul 08 18:27:03.293102 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:03.293044 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" podStartSLOduration=2.571414298 podStartE2EDuration="7.293029038s" totalImagesPullingTime="4.72161474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:27:03.291787671 +0000 UTC m=+557.915374752" watchObservedRunningTime="2026-07-08 18:27:03.293029038 +0000 UTC m=+557.916616118" Jul 08 18:27:04.162326 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:04.162299 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:27:07.232386 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.232353 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx"] Jul 08 18:27:07.236320 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.236299 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:07.240188 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.240165 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 18:27:07.242109 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.241546 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-kgx6p\"" Jul 08 18:27:07.242109 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.244604 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn"] Jul 08 18:27:07.251918 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.251900 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx"] Jul 08 18:27:07.252070 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.252017 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:07.254485 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.254467 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 08 18:27:07.254579 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.254531 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hkcxv\"" Jul 08 18:27:07.266564 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.266544 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn"] Jul 08 18:27:07.322868 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.322841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stcq\" (UniqueName: \"kubernetes.io/projected/ac29eed7-6326-42d7-8e18-26e56809718e-kube-api-access-9stcq\") pod \"llmisvc-controller-manager-777c6bd7d5-vjldn\" (UID: \"ac29eed7-6326-42d7-8e18-26e56809718e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:07.322999 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.322895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert\") pod \"kserve-controller-manager-d6f4b65b8-c8fwx\" (UID: \"7438f9bd-2911-460e-afe2-e2f4abfc2b00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:07.322999 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.322943 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcjs\" (UniqueName: \"kubernetes.io/projected/7438f9bd-2911-460e-afe2-e2f4abfc2b00-kube-api-access-rfcjs\") pod \"kserve-controller-manager-d6f4b65b8-c8fwx\" (UID: \"7438f9bd-2911-460e-afe2-e2f4abfc2b00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:07.322999 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.322975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vjldn\" (UID: \"ac29eed7-6326-42d7-8e18-26e56809718e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:07.423876 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.423852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stcq\" (UniqueName: \"kubernetes.io/projected/ac29eed7-6326-42d7-8e18-26e56809718e-kube-api-access-9stcq\") pod \"llmisvc-controller-manager-777c6bd7d5-vjldn\" (UID: \"ac29eed7-6326-42d7-8e18-26e56809718e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:07.423974 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.423901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert\") pod \"kserve-controller-manager-d6f4b65b8-c8fwx\" (UID: \"7438f9bd-2911-460e-afe2-e2f4abfc2b00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:07.423974 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.423938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcjs\" (UniqueName: \"kubernetes.io/projected/7438f9bd-2911-460e-afe2-e2f4abfc2b00-kube-api-access-rfcjs\") pod \"kserve-controller-manager-d6f4b65b8-c8fwx\" (UID: \"7438f9bd-2911-460e-afe2-e2f4abfc2b00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:07.424066 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.423976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vjldn\" (UID: \"ac29eed7-6326-42d7-8e18-26e56809718e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:07.424124 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:07.424107 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 08 18:27:07.424176 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:07.424155 2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 08 18:27:07.424228 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:07.424160 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert podName:ac29eed7-6326-42d7-8e18-26e56809718e nodeName:}" failed. No retries permitted until 2026-07-08 18:27:07.924142428 +0000 UTC m=+562.547729485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert") pod "llmisvc-controller-manager-777c6bd7d5-vjldn" (UID: "ac29eed7-6326-42d7-8e18-26e56809718e") : secret "llmisvc-webhook-server-cert" not found Jul 08 18:27:07.424283 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:07.424241 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert podName:7438f9bd-2911-460e-afe2-e2f4abfc2b00 nodeName:}" failed. No retries permitted until 2026-07-08 18:27:07.924224347 +0000 UTC m=+562.547811421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert") pod "kserve-controller-manager-d6f4b65b8-c8fwx" (UID: "7438f9bd-2911-460e-afe2-e2f4abfc2b00") : secret "kserve-webhook-server-cert" not found Jul 08 18:27:07.432850 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.432826 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcjs\" (UniqueName: \"kubernetes.io/projected/7438f9bd-2911-460e-afe2-e2f4abfc2b00-kube-api-access-rfcjs\") pod \"kserve-controller-manager-d6f4b65b8-c8fwx\" (UID: \"7438f9bd-2911-460e-afe2-e2f4abfc2b00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:07.433132 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.433120 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stcq\" (UniqueName: \"kubernetes.io/projected/ac29eed7-6326-42d7-8e18-26e56809718e-kube-api-access-9stcq\") pod \"llmisvc-controller-manager-777c6bd7d5-vjldn\" (UID: \"ac29eed7-6326-42d7-8e18-26e56809718e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:07.927992 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.927960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert\") pod \"kserve-controller-manager-d6f4b65b8-c8fwx\" (UID: \"7438f9bd-2911-460e-afe2-e2f4abfc2b00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:07.928151 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:07.928005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vjldn\" (UID: \"ac29eed7-6326-42d7-8e18-26e56809718e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:07.928151 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:07.928096 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 08 18:27:07.928151 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:07.928111 2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 08 18:27:07.928151 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:07.928142 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert podName:ac29eed7-6326-42d7-8e18-26e56809718e nodeName:}" failed. No retries permitted until 2026-07-08 18:27:08.928129601 +0000 UTC m=+563.551716659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert") pod "llmisvc-controller-manager-777c6bd7d5-vjldn" (UID: "ac29eed7-6326-42d7-8e18-26e56809718e") : secret "llmisvc-webhook-server-cert" not found Jul 08 18:27:07.928299 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:07.928172 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert podName:7438f9bd-2911-460e-afe2-e2f4abfc2b00 nodeName:}" failed. No retries permitted until 2026-07-08 18:27:08.928155912 +0000 UTC m=+563.551742972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert") pod "kserve-controller-manager-d6f4b65b8-c8fwx" (UID: "7438f9bd-2911-460e-afe2-e2f4abfc2b00") : secret "kserve-webhook-server-cert" not found Jul 08 18:27:08.936814 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:08.936779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert\") pod \"kserve-controller-manager-d6f4b65b8-c8fwx\" (UID: \"7438f9bd-2911-460e-afe2-e2f4abfc2b00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:08.937196 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:08.936827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vjldn\" (UID: \"ac29eed7-6326-42d7-8e18-26e56809718e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:08.939213 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:08.939194 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac29eed7-6326-42d7-8e18-26e56809718e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vjldn\" (UID: \"ac29eed7-6326-42d7-8e18-26e56809718e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:08.939304 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:08.939287 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7438f9bd-2911-460e-afe2-e2f4abfc2b00-cert\") pod \"kserve-controller-manager-d6f4b65b8-c8fwx\" (UID: \"7438f9bd-2911-460e-afe2-e2f4abfc2b00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:09.052601 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:09.052571 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:09.069254 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:09.069227 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:09.202327 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:09.202302 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx"] Jul 08 18:27:09.204561 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:27:09.204529 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7438f9bd_2911_460e_afe2_e2f4abfc2b00.slice/crio-9162ec35e7858dd177fc6e5c3ccd5d0066bcb447e7a7f7b480d7355f35f10ca9 WatchSource:0}: Error finding container 9162ec35e7858dd177fc6e5c3ccd5d0066bcb447e7a7f7b480d7355f35f10ca9: Status 404 returned error can't find the container with id 9162ec35e7858dd177fc6e5c3ccd5d0066bcb447e7a7f7b480d7355f35f10ca9 Jul 08 18:27:09.222244 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:09.222220 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn"] Jul 08 18:27:09.223735 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:27:09.223713 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podac29eed7_6326_42d7_8e18_26e56809718e.slice/crio-5bea134a89438c3163c46d4792b1c3d11d3263109c6aa223e4bd589c563b0b4c WatchSource:0}: Error finding container 5bea134a89438c3163c46d4792b1c3d11d3263109c6aa223e4bd589c563b0b4c: Status 404 returned error can't find the container with id 5bea134a89438c3163c46d4792b1c3d11d3263109c6aa223e4bd589c563b0b4c Jul 08 18:27:10.199423 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:10.199390 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" event={"ID":"ac29eed7-6326-42d7-8e18-26e56809718e","Type":"ContainerStarted","Data":"5bea134a89438c3163c46d4792b1c3d11d3263109c6aa223e4bd589c563b0b4c"} Jul 08 18:27:10.200645 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:10.200618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" event={"ID":"7438f9bd-2911-460e-afe2-e2f4abfc2b00","Type":"ContainerStarted","Data":"9162ec35e7858dd177fc6e5c3ccd5d0066bcb447e7a7f7b480d7355f35f10ca9"} Jul 08 18:27:10.802514 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:10.802480 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sx74s"] Jul 08 18:27:10.802740 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:10.802715 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" podUID="7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" containerName="discovery" containerID="cri-o://2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912" gracePeriod=30 Jul 08 18:27:11.080986 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.080962 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:27:11.163101 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.163069 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-dns-cert\") pod \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " Jul 08 18:27:11.163273 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.163121 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-kubeconfig\") pod \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " Jul 08 18:27:11.163273 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.163153 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-token\") pod \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " Jul 08 18:27:11.163273 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.163206 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-kube-api-access-9b9ns\") pod \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " Jul 08 18:27:11.163273 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.163231 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-cacerts\") pod \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " Jul 08 18:27:11.163482 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.163311 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-local-certs\") pod \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " Jul 08 18:27:11.163482 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.163345 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-ca-configmap\") pod \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\" (UID: \"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e\") " Jul 08 18:27:11.163927 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.163877 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" (UID: "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:27:11.165965 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.165934 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-local-certs" (OuterVolumeSpecName: "local-certs") pod "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" (UID: "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:27:11.166081 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.166030 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-kube-api-access-9b9ns" (OuterVolumeSpecName: "kube-api-access-9b9ns") pod "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" (UID: "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e"). InnerVolumeSpecName "kube-api-access-9b9ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:27:11.166151 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.166130 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" (UID: "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:27:11.166549 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.166513 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" (UID: "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:27:11.166749 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.166708 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-cacerts" (OuterVolumeSpecName: "cacerts") pod "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" (UID: "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:27:11.167253 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.167232 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" (UID: "7fa4b05e-bfb6-4ce1-b463-012e137b1d5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:27:11.209341 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.209312 2565 generic.go:356] "Generic (PLEG): container finished" podID="7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" containerID="2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912" exitCode=0 Jul 08 18:27:11.209754 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.209363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" event={"ID":"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e","Type":"ContainerDied","Data":"2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912"} Jul 08 18:27:11.209754 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.209369 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" Jul 08 18:27:11.209754 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.209402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-sx74s" event={"ID":"7fa4b05e-bfb6-4ce1-b463-012e137b1d5e","Type":"ContainerDied","Data":"1b5dc3b1523a048256338fa6d33186040ecbfd482033235eddf749a5706e4cb9"} Jul 08 18:27:11.209754 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.209427 2565 scope.go:117] "RemoveContainer" containerID="2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912" Jul 08 18:27:11.221194 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.221174 2565 scope.go:117] "RemoveContainer" containerID="2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912" Jul 08 18:27:11.221486 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:27:11.221451 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912\": container with ID starting with 2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912 not found: ID does not exist" containerID="2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912" Jul 08 18:27:11.221566 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.221498 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912"} err="failed to get container status \"2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912\": rpc error: code = NotFound desc = could not find container \"2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912\": container with ID starting with 2540f70daf4bfe76ace5a2c91e832910443eb288b78793a3444960db2db5e912 not found: ID does not exist" Jul 08 18:27:11.235501 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.235482 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sx74s"] Jul 08 18:27:11.239327 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.239305 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-sx74s"] Jul 08 18:27:11.264708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.264687 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-local-certs\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:27:11.264708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.264706 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-ca-configmap\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:27:11.264808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.264714 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-csr-dns-cert\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:27:11.264808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.264723 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-kubeconfig\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:27:11.264808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.264731 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-istio-token\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:27:11.264808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.264739 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-kube-api-access-9b9ns\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:27:11.264808 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.264746 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e-cacerts\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:27:11.930293 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:11.930245 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" path="/var/lib/kubelet/pods/7fa4b05e-bfb6-4ce1-b463-012e137b1d5e/volumes" Jul 08 18:27:13.222871 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:13.222783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" event={"ID":"ac29eed7-6326-42d7-8e18-26e56809718e","Type":"ContainerStarted","Data":"76306fe579460613fd4729d6fac900548a7ab6db51e0279dcc508ecabab36260"} Jul 08 18:27:13.223275 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:13.222893 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:13.224586 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:13.224555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" event={"ID":"7438f9bd-2911-460e-afe2-e2f4abfc2b00","Type":"ContainerStarted","Data":"99e0a42c2dbac60609cdd1ff69e671dfb7bca6a16631b3bf6d60d34459eb9896"} Jul 08 18:27:13.224719 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:13.224684 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:13.245810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:13.245767 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" podStartSLOduration=2.651145701 podStartE2EDuration="6.24574239s" totalImagesPullingTime="3.594596689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:27:13.245528331 +0000 UTC m=+567.869115411" watchObservedRunningTime="2026-07-08 18:27:13.24574239 +0000 UTC m=+567.869329471" Jul 08 18:27:13.266963 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:13.266926 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" podStartSLOduration=2.659325569 podStartE2EDuration="6.26691424s" totalImagesPullingTime="3.607588671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:27:13.265218105 +0000 UTC m=+567.888805178" watchObservedRunningTime="2026-07-08 18:27:13.26691424 +0000 UTC m=+567.890501320" Jul 08 18:27:14.163297 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:14.163270 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-tlmk4" Jul 08 18:27:15.174716 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:15.174687 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-x59v7" Jul 08 18:27:44.232543 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:44.232512 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vjldn" Jul 08 18:27:44.235388 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:44.235372 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c8fwx" Jul 08 18:27:45.750719 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:45.750689 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:27:45.751631 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:45.751608 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:27:45.753240 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:45.753227 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:27:45.753987 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:45.753960 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:27:56.742429 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.742395 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2t74p"] Jul 08 18:27:56.743966 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.742911 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" containerName="discovery" Jul 08 18:27:56.743966 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.742926 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" containerName="discovery" Jul 08 18:27:56.743966 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.743024 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa4b05e-bfb6-4ce1-b463-012e137b1d5e" containerName="discovery" Jul 08 18:27:56.744936 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.744910 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" Jul 08 18:27:56.747679 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.747643 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-wxsxp\"" Jul 08 18:27:56.747805 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.747708 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 18:27:56.748553 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.748531 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 18:27:56.754793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.754681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2t74p"] Jul 08 18:27:56.911921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:56.911892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2pqw\" (UniqueName: \"kubernetes.io/projected/ec593e10-87cf-443d-a39d-cc2915324c22-kube-api-access-x2pqw\") pod \"kuadrant-operator-catalog-2t74p\" (UID: \"ec593e10-87cf-443d-a39d-cc2915324c22\") " pod="kuadrant-system/kuadrant-operator-catalog-2t74p" Jul 08 18:27:57.012290 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.012236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2pqw\" (UniqueName: \"kubernetes.io/projected/ec593e10-87cf-443d-a39d-cc2915324c22-kube-api-access-x2pqw\") pod \"kuadrant-operator-catalog-2t74p\" (UID: \"ec593e10-87cf-443d-a39d-cc2915324c22\") " pod="kuadrant-system/kuadrant-operator-catalog-2t74p" Jul 08 18:27:57.020078 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.020058 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2pqw\" (UniqueName: \"kubernetes.io/projected/ec593e10-87cf-443d-a39d-cc2915324c22-kube-api-access-x2pqw\") pod \"kuadrant-operator-catalog-2t74p\" (UID: \"ec593e10-87cf-443d-a39d-cc2915324c22\") " pod="kuadrant-system/kuadrant-operator-catalog-2t74p" Jul 08 18:27:57.055633 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.055610 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" Jul 08 18:27:57.101894 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.101870 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2t74p"] Jul 08 18:27:57.188714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.188689 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2t74p"] Jul 08 18:27:57.190067 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:27:57.190038 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec593e10_87cf_443d_a39d_cc2915324c22.slice/crio-3ac08d7e8193134b78d17f261afe0a1e7c03831d7b6efce464efed87cddc7258 WatchSource:0}: Error finding container 3ac08d7e8193134b78d17f261afe0a1e7c03831d7b6efce464efed87cddc7258: Status 404 returned error can't find the container with id 3ac08d7e8193134b78d17f261afe0a1e7c03831d7b6efce464efed87cddc7258 Jul 08 18:27:57.191438 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.191422 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:27:57.310786 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.310736 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mx9zk"] Jul 08 18:27:57.313471 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.313456 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:27:57.322034 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.322015 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mx9zk"] Jul 08 18:27:57.414828 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.414801 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxkp9\" (UniqueName: \"kubernetes.io/projected/f9041d2e-f065-4c18-b340-0b2b47779f41-kube-api-access-vxkp9\") pod \"kuadrant-operator-catalog-mx9zk\" (UID: \"f9041d2e-f065-4c18-b340-0b2b47779f41\") " pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:27:57.442687 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.442644 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" event={"ID":"ec593e10-87cf-443d-a39d-cc2915324c22","Type":"ContainerStarted","Data":"3ac08d7e8193134b78d17f261afe0a1e7c03831d7b6efce464efed87cddc7258"} Jul 08 18:27:57.515960 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.515931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxkp9\" (UniqueName: \"kubernetes.io/projected/f9041d2e-f065-4c18-b340-0b2b47779f41-kube-api-access-vxkp9\") pod \"kuadrant-operator-catalog-mx9zk\" (UID: \"f9041d2e-f065-4c18-b340-0b2b47779f41\") " pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:27:57.524016 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.523991 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxkp9\" (UniqueName: \"kubernetes.io/projected/f9041d2e-f065-4c18-b340-0b2b47779f41-kube-api-access-vxkp9\") pod \"kuadrant-operator-catalog-mx9zk\" (UID: \"f9041d2e-f065-4c18-b340-0b2b47779f41\") " pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:27:57.623481 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.623430 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:27:57.751176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:57.751152 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mx9zk"] Jul 08 18:27:57.752948 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:27:57.752922 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9041d2e_f065_4c18_b340_0b2b47779f41.slice/crio-b443651ffe83560ff57941d7b6ea187334982592800b30c4d022774c4919cb41 WatchSource:0}: Error finding container b443651ffe83560ff57941d7b6ea187334982592800b30c4d022774c4919cb41: Status 404 returned error can't find the container with id b443651ffe83560ff57941d7b6ea187334982592800b30c4d022774c4919cb41 Jul 08 18:27:58.451778 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:27:58.451745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" event={"ID":"f9041d2e-f065-4c18-b340-0b2b47779f41","Type":"ContainerStarted","Data":"b443651ffe83560ff57941d7b6ea187334982592800b30c4d022774c4919cb41"} Jul 08 18:28:00.463380 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.463344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" event={"ID":"f9041d2e-f065-4c18-b340-0b2b47779f41","Type":"ContainerStarted","Data":"0c839aaff7c992a9ed4f714f72cedae13db93fbefd6f9bab4fd5b6cf032ea1bf"} Jul 08 18:28:00.464720 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.464694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" event={"ID":"ec593e10-87cf-443d-a39d-cc2915324c22","Type":"ContainerStarted","Data":"132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62"} Jul 08 18:28:00.464830 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.464765 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" podUID="ec593e10-87cf-443d-a39d-cc2915324c22" containerName="registry-server" containerID="cri-o://132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62" gracePeriod=2 Jul 08 18:28:00.479297 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.479251 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" podStartSLOduration=1.821801644 podStartE2EDuration="3.479235111s" totalImagesPullingTime="1.657433467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:00.477568874 +0000 UTC m=+615.101155954" watchObservedRunningTime="2026-07-08 18:28:00.479235111 +0000 UTC m=+615.102822190" Jul 08 18:28:00.493417 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.493379 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" podStartSLOduration=2.276234923 podStartE2EDuration="4.493367495s" totalImagesPullingTime="2.217132572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:00.492000231 +0000 UTC m=+615.115587313" watchObservedRunningTime="2026-07-08 18:28:00.493367495 +0000 UTC m=+615.116954574" Jul 08 18:28:00.704116 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.704092 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" Jul 08 18:28:00.845842 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.845818 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2pqw\" (UniqueName: \"kubernetes.io/projected/ec593e10-87cf-443d-a39d-cc2915324c22-kube-api-access-x2pqw\") pod \"ec593e10-87cf-443d-a39d-cc2915324c22\" (UID: \"ec593e10-87cf-443d-a39d-cc2915324c22\") " Jul 08 18:28:00.847922 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.847899 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec593e10-87cf-443d-a39d-cc2915324c22-kube-api-access-x2pqw" (OuterVolumeSpecName: "kube-api-access-x2pqw") pod "ec593e10-87cf-443d-a39d-cc2915324c22" (UID: "ec593e10-87cf-443d-a39d-cc2915324c22"). InnerVolumeSpecName "kube-api-access-x2pqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:28:00.946257 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:00.946237 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2pqw\" (UniqueName: \"kubernetes.io/projected/ec593e10-87cf-443d-a39d-cc2915324c22-kube-api-access-x2pqw\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:01.473383 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.473355 2565 generic.go:356] "Generic (PLEG): container finished" podID="ec593e10-87cf-443d-a39d-cc2915324c22" containerID="132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62" exitCode=0 Jul 08 18:28:01.473987 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.473396 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" Jul 08 18:28:01.473987 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.473450 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" event={"ID":"ec593e10-87cf-443d-a39d-cc2915324c22","Type":"ContainerDied","Data":"132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62"} Jul 08 18:28:01.473987 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.473507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2t74p" event={"ID":"ec593e10-87cf-443d-a39d-cc2915324c22","Type":"ContainerDied","Data":"3ac08d7e8193134b78d17f261afe0a1e7c03831d7b6efce464efed87cddc7258"} Jul 08 18:28:01.473987 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.473538 2565 scope.go:117] "RemoveContainer" containerID="132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62" Jul 08 18:28:01.483331 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.483315 2565 scope.go:117] "RemoveContainer" containerID="132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62" Jul 08 18:28:01.483593 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:01.483575 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62\": container with ID starting with 132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62 not found: ID does not exist" containerID="132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62" Jul 08 18:28:01.483672 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.483603 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62"} err="failed to get container status \"132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62\": rpc error: code = NotFound desc = could not find container \"132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62\": container with ID starting with 132b24bf80fce4728b6a65fcb9d091f3436dbd021c7c546f2c4e0231aa22aa62 not found: ID does not exist" Jul 08 18:28:01.494565 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.494545 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2t74p"] Jul 08 18:28:01.498871 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.498854 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2t74p"] Jul 08 18:28:01.927382 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:01.927363 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec593e10-87cf-443d-a39d-cc2915324c22" path="/var/lib/kubelet/pods/ec593e10-87cf-443d-a39d-cc2915324c22/volumes" Jul 08 18:28:07.623951 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:07.623924 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:28:07.624352 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:07.623970 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:28:07.646288 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:07.646266 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:28:08.532560 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:08.532537 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-mx9zk" Jul 08 18:28:12.114766 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.114734 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p"] Jul 08 18:28:12.115127 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.115117 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec593e10-87cf-443d-a39d-cc2915324c22" containerName="registry-server" Jul 08 18:28:12.115175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.115129 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec593e10-87cf-443d-a39d-cc2915324c22" containerName="registry-server" Jul 08 18:28:12.115211 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.115194 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec593e10-87cf-443d-a39d-cc2915324c22" containerName="registry-server" Jul 08 18:28:12.119931 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.119883 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.123131 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.123107 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kg9gr\"" Jul 08 18:28:12.127382 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.127362 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p"] Jul 08 18:28:12.230313 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.230286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.230474 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.230331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45xrw\" (UniqueName: \"kubernetes.io/projected/57886b32-8a1a-4cce-8729-dff55986dcca-kube-api-access-45xrw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.230474 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.230355 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.331065 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.331042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.331160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.331077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45xrw\" (UniqueName: \"kubernetes.io/projected/57886b32-8a1a-4cce-8729-dff55986dcca-kube-api-access-45xrw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.331160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.331097 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.331397 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.331382 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.331434 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.331411 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.340170 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.340140 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45xrw\" (UniqueName: \"kubernetes.io/projected/57886b32-8a1a-4cce-8729-dff55986dcca-kube-api-access-45xrw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.430417 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.430369 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:12.564176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.564150 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p"] Jul 08 18:28:12.565226 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:12.565199 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57886b32_8a1a_4cce_8729_dff55986dcca.slice/crio-d62a33f3a1d44a57abde2fcbdcc28edd7f4c9daf6abac56c64bd46bab1a292ec WatchSource:0}: Error finding container d62a33f3a1d44a57abde2fcbdcc28edd7f4c9daf6abac56c64bd46bab1a292ec: Status 404 returned error can't find the container with id d62a33f3a1d44a57abde2fcbdcc28edd7f4c9daf6abac56c64bd46bab1a292ec Jul 08 18:28:12.715856 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.715800 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj"] Jul 08 18:28:12.719327 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.719310 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.727532 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.727512 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj"] Jul 08 18:28:12.834954 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.834934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.835087 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.835013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.835087 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.835039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjdh\" (UniqueName: \"kubernetes.io/projected/2db70365-7b9e-4e32-9c87-05bf0c316ccb-kube-api-access-nwjdh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.935337 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.935311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.935436 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.935342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjdh\" (UniqueName: \"kubernetes.io/projected/2db70365-7b9e-4e32-9c87-05bf0c316ccb-kube-api-access-nwjdh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.935436 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.935371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.935647 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.935627 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.935747 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.935732 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:12.943673 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:12.943648 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjdh\" (UniqueName: \"kubernetes.io/projected/2db70365-7b9e-4e32-9c87-05bf0c316ccb-kube-api-access-nwjdh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:13.029721 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.029697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:13.117030 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.116707 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j"] Jul 08 18:28:13.123435 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.123410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.131859 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.131833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j"] Jul 08 18:28:13.172883 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.172859 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj"] Jul 08 18:28:13.178191 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:13.178164 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db70365_7b9e_4e32_9c87_05bf0c316ccb.slice/crio-ab07219569775e2c92308d74636040f87d7d0459f8e6df275e942f855b83a353 WatchSource:0}: Error finding container ab07219569775e2c92308d74636040f87d7d0459f8e6df275e942f855b83a353: Status 404 returned error can't find the container with id ab07219569775e2c92308d74636040f87d7d0459f8e6df275e942f855b83a353 Jul 08 18:28:13.237349 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.237327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.237451 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.237359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.237451 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.237382 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgkl\" (UniqueName: \"kubernetes.io/projected/2f2007a1-b0d4-4579-9875-c29531b12604-kube-api-access-tcgkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.338537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.338476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.338537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.338506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.338537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.338527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgkl\" (UniqueName: \"kubernetes.io/projected/2f2007a1-b0d4-4579-9875-c29531b12604-kube-api-access-tcgkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.338842 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.338822 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.338911 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.338893 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.346864 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.346839 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgkl\" (UniqueName: \"kubernetes.io/projected/2f2007a1-b0d4-4579-9875-c29531b12604-kube-api-access-tcgkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.435728 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.435708 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:13.522002 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.521973 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp"] Jul 08 18:28:13.530797 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.530776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.536526 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.536092 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp"] Jul 08 18:28:13.540380 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.538483 2565 generic.go:356] "Generic (PLEG): container finished" podID="57886b32-8a1a-4cce-8729-dff55986dcca" containerID="41548d93313a7a6d92ab9bdc24ed70ca3a8b9a84c935ef851736041a4edff935" exitCode=0 Jul 08 18:28:13.540380 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.538607 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" event={"ID":"57886b32-8a1a-4cce-8729-dff55986dcca","Type":"ContainerDied","Data":"41548d93313a7a6d92ab9bdc24ed70ca3a8b9a84c935ef851736041a4edff935"} Jul 08 18:28:13.540380 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.538646 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" event={"ID":"57886b32-8a1a-4cce-8729-dff55986dcca","Type":"ContainerStarted","Data":"d62a33f3a1d44a57abde2fcbdcc28edd7f4c9daf6abac56c64bd46bab1a292ec"} Jul 08 18:28:13.541639 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.541443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.541639 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.541496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.541785 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.541636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gsq5\" (UniqueName: \"kubernetes.io/projected/52021da0-e11d-41eb-846a-f1074a6639c4-kube-api-access-2gsq5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.543248 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.543226 2565 generic.go:356] "Generic (PLEG): container finished" podID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerID="c96df787ba51fe2275f1fee21fcae00585a9dd048a0fe2ffaf73cc4671669595" exitCode=0 Jul 08 18:28:13.543347 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.543259 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" event={"ID":"2db70365-7b9e-4e32-9c87-05bf0c316ccb","Type":"ContainerDied","Data":"c96df787ba51fe2275f1fee21fcae00585a9dd048a0fe2ffaf73cc4671669595"} Jul 08 18:28:13.543347 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.543291 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" event={"ID":"2db70365-7b9e-4e32-9c87-05bf0c316ccb","Type":"ContainerStarted","Data":"ab07219569775e2c92308d74636040f87d7d0459f8e6df275e942f855b83a353"} Jul 08 18:28:13.568187 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.568169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j"] Jul 08 18:28:13.569914 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:13.569890 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2007a1_b0d4_4579_9875_c29531b12604.slice/crio-d9315bea9e817b881caf8c3487c9ba2c18f3d34764ee0564213530338cea319e WatchSource:0}: Error finding container d9315bea9e817b881caf8c3487c9ba2c18f3d34764ee0564213530338cea319e: Status 404 returned error can't find the container with id d9315bea9e817b881caf8c3487c9ba2c18f3d34764ee0564213530338cea319e Jul 08 18:28:13.642017 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.641995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.642106 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.642033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.642175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.642162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gsq5\" (UniqueName: \"kubernetes.io/projected/52021da0-e11d-41eb-846a-f1074a6639c4-kube-api-access-2gsq5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.642391 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.642370 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.642464 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.642446 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.655278 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.655255 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gsq5\" (UniqueName: \"kubernetes.io/projected/52021da0-e11d-41eb-846a-f1074a6639c4-kube-api-access-2gsq5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.844618 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.844596 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:13.973258 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:13.973236 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp"] Jul 08 18:28:13.974715 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:13.974689 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52021da0_e11d_41eb_846a_f1074a6639c4.slice/crio-3e0b8de673682e7fdc3624d4da18928b84547ad2b67206db11580851f0cd6460 WatchSource:0}: Error finding container 3e0b8de673682e7fdc3624d4da18928b84547ad2b67206db11580851f0cd6460: Status 404 returned error can't find the container with id 3e0b8de673682e7fdc3624d4da18928b84547ad2b67206db11580851f0cd6460 Jul 08 18:28:14.551906 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.551882 2565 generic.go:356] "Generic (PLEG): container finished" podID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerID="0457bc117e91fa4b7c2deafd7709565088f643a236910de43d363d47629a8861" exitCode=0 Jul 08 18:28:14.552274 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.551931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" event={"ID":"2db70365-7b9e-4e32-9c87-05bf0c316ccb","Type":"ContainerDied","Data":"0457bc117e91fa4b7c2deafd7709565088f643a236910de43d363d47629a8861"} Jul 08 18:28:14.554389 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.554359 2565 generic.go:356] "Generic (PLEG): container finished" podID="52021da0-e11d-41eb-846a-f1074a6639c4" containerID="54f6841a3b48c422824d7ef7d2f3472d74bf60dfbb42db5bf86e8e2235b237bc" exitCode=0 Jul 08 18:28:14.554845 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.554604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" event={"ID":"52021da0-e11d-41eb-846a-f1074a6639c4","Type":"ContainerDied","Data":"54f6841a3b48c422824d7ef7d2f3472d74bf60dfbb42db5bf86e8e2235b237bc"} Jul 08 18:28:14.554845 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.554698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" event={"ID":"52021da0-e11d-41eb-846a-f1074a6639c4","Type":"ContainerStarted","Data":"3e0b8de673682e7fdc3624d4da18928b84547ad2b67206db11580851f0cd6460"} Jul 08 18:28:14.557916 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.557891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" event={"ID":"57886b32-8a1a-4cce-8729-dff55986dcca","Type":"ContainerStarted","Data":"e6e28b2e8271787ef2bce78017ef3079486a638173bcd153ad68af0a504d79a6"} Jul 08 18:28:14.563930 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.563906 2565 generic.go:356] "Generic (PLEG): container finished" podID="2f2007a1-b0d4-4579-9875-c29531b12604" containerID="ba5e4d4d971d43e1f14489037a831f514258e1e276bd1ce7ba3dbbcafc0a74e2" exitCode=0 Jul 08 18:28:14.564007 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.563953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" event={"ID":"2f2007a1-b0d4-4579-9875-c29531b12604","Type":"ContainerDied","Data":"ba5e4d4d971d43e1f14489037a831f514258e1e276bd1ce7ba3dbbcafc0a74e2"} Jul 08 18:28:14.564007 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:14.563985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" event={"ID":"2f2007a1-b0d4-4579-9875-c29531b12604","Type":"ContainerStarted","Data":"d9315bea9e817b881caf8c3487c9ba2c18f3d34764ee0564213530338cea319e"} Jul 08 18:28:15.571613 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:15.571584 2565 generic.go:356] "Generic (PLEG): container finished" podID="57886b32-8a1a-4cce-8729-dff55986dcca" containerID="e6e28b2e8271787ef2bce78017ef3079486a638173bcd153ad68af0a504d79a6" exitCode=0 Jul 08 18:28:15.572018 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:15.571640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" event={"ID":"57886b32-8a1a-4cce-8729-dff55986dcca","Type":"ContainerDied","Data":"e6e28b2e8271787ef2bce78017ef3079486a638173bcd153ad68af0a504d79a6"} Jul 08 18:28:15.573750 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:15.573731 2565 generic.go:356] "Generic (PLEG): container finished" podID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerID="0972ab4413596b4c558a1992bf4df02667e95d75a6eb9dc4bb7888d974ab19fa" exitCode=0 Jul 08 18:28:15.573866 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:15.573817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" event={"ID":"2db70365-7b9e-4e32-9c87-05bf0c316ccb","Type":"ContainerDied","Data":"0972ab4413596b4c558a1992bf4df02667e95d75a6eb9dc4bb7888d974ab19fa"} Jul 08 18:28:15.575676 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:15.575638 2565 generic.go:356] "Generic (PLEG): container finished" podID="52021da0-e11d-41eb-846a-f1074a6639c4" containerID="376dd0250e4502e10d9e7f333e1106789ee6373bb01c49e68bde3f5bcf9e8abd" exitCode=0 Jul 08 18:28:15.575774 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:15.575690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" event={"ID":"52021da0-e11d-41eb-846a-f1074a6639c4","Type":"ContainerDied","Data":"376dd0250e4502e10d9e7f333e1106789ee6373bb01c49e68bde3f5bcf9e8abd"} Jul 08 18:28:16.582870 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.582840 2565 generic.go:356] "Generic (PLEG): container finished" podID="2f2007a1-b0d4-4579-9875-c29531b12604" containerID="9f32e75c2f799f391db8b89b1ff5ebad3554b1de18985bfa5cdbfec9164e054a" exitCode=0 Jul 08 18:28:16.583296 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.582939 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" event={"ID":"2f2007a1-b0d4-4579-9875-c29531b12604","Type":"ContainerDied","Data":"9f32e75c2f799f391db8b89b1ff5ebad3554b1de18985bfa5cdbfec9164e054a"} Jul 08 18:28:16.585157 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.585132 2565 generic.go:356] "Generic (PLEG): container finished" podID="52021da0-e11d-41eb-846a-f1074a6639c4" containerID="5408b3f49f48602f93607373329ce6daa9746d86da0e09d3878f6c4739804b64" exitCode=0 Jul 08 18:28:16.585254 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.585160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" event={"ID":"52021da0-e11d-41eb-846a-f1074a6639c4","Type":"ContainerDied","Data":"5408b3f49f48602f93607373329ce6daa9746d86da0e09d3878f6c4739804b64"} Jul 08 18:28:16.587264 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.587246 2565 generic.go:356] "Generic (PLEG): container finished" podID="57886b32-8a1a-4cce-8729-dff55986dcca" containerID="b7b2b7dd7208a3b541f154b5d1b219e5f289679a3cb57ce78f0c3f1e55a5dcd6" exitCode=0 Jul 08 18:28:16.587362 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.587324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" event={"ID":"57886b32-8a1a-4cce-8729-dff55986dcca","Type":"ContainerDied","Data":"b7b2b7dd7208a3b541f154b5d1b219e5f289679a3cb57ce78f0c3f1e55a5dcd6"} Jul 08 18:28:16.741206 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.741184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:16.767896 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.767874 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-util\") pod \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " Jul 08 18:28:16.768029 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.767914 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwjdh\" (UniqueName: \"kubernetes.io/projected/2db70365-7b9e-4e32-9c87-05bf0c316ccb-kube-api-access-nwjdh\") pod \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " Jul 08 18:28:16.768029 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.768014 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-bundle\") pod \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\" (UID: \"2db70365-7b9e-4e32-9c87-05bf0c316ccb\") " Jul 08 18:28:16.768417 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.768395 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-bundle" (OuterVolumeSpecName: "bundle") pod "2db70365-7b9e-4e32-9c87-05bf0c316ccb" (UID: "2db70365-7b9e-4e32-9c87-05bf0c316ccb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:16.770284 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.770259 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db70365-7b9e-4e32-9c87-05bf0c316ccb-kube-api-access-nwjdh" (OuterVolumeSpecName: "kube-api-access-nwjdh") pod "2db70365-7b9e-4e32-9c87-05bf0c316ccb" (UID: "2db70365-7b9e-4e32-9c87-05bf0c316ccb"). InnerVolumeSpecName "kube-api-access-nwjdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:28:16.772797 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.772774 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-util" (OuterVolumeSpecName: "util") pod "2db70365-7b9e-4e32-9c87-05bf0c316ccb" (UID: "2db70365-7b9e-4e32-9c87-05bf0c316ccb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:16.869064 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.869041 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-bundle\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:16.869064 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.869061 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db70365-7b9e-4e32-9c87-05bf0c316ccb-util\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:16.869182 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:16.869070 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwjdh\" (UniqueName: \"kubernetes.io/projected/2db70365-7b9e-4e32-9c87-05bf0c316ccb-kube-api-access-nwjdh\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:17.594850 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.594825 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" Jul 08 18:28:17.594850 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.594840 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj" event={"ID":"2db70365-7b9e-4e32-9c87-05bf0c316ccb","Type":"ContainerDied","Data":"ab07219569775e2c92308d74636040f87d7d0459f8e6df275e942f855b83a353"} Jul 08 18:28:17.595235 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.594907 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab07219569775e2c92308d74636040f87d7d0459f8e6df275e942f855b83a353" Jul 08 18:28:17.597100 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.597080 2565 generic.go:356] "Generic (PLEG): container finished" podID="2f2007a1-b0d4-4579-9875-c29531b12604" containerID="9a467a6a8cbc628c5e27b3aadc9af2632f6491b66dec66b5ed1c9c73ecb32649" exitCode=0 Jul 08 18:28:17.597231 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.597112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" event={"ID":"2f2007a1-b0d4-4579-9875-c29531b12604","Type":"ContainerDied","Data":"9a467a6a8cbc628c5e27b3aadc9af2632f6491b66dec66b5ed1c9c73ecb32649"} Jul 08 18:28:17.753886 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.753862 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:17.756396 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.756379 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:17.783769 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.783748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-util\") pod \"57886b32-8a1a-4cce-8729-dff55986dcca\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " Jul 08 18:28:17.783861 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.783799 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-util\") pod \"52021da0-e11d-41eb-846a-f1074a6639c4\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " Jul 08 18:28:17.783861 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.783834 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-bundle\") pod \"57886b32-8a1a-4cce-8729-dff55986dcca\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " Jul 08 18:28:17.783941 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.783864 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gsq5\" (UniqueName: \"kubernetes.io/projected/52021da0-e11d-41eb-846a-f1074a6639c4-kube-api-access-2gsq5\") pod \"52021da0-e11d-41eb-846a-f1074a6639c4\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " Jul 08 18:28:17.783986 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.783962 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45xrw\" (UniqueName: \"kubernetes.io/projected/57886b32-8a1a-4cce-8729-dff55986dcca-kube-api-access-45xrw\") pod \"57886b32-8a1a-4cce-8729-dff55986dcca\" (UID: \"57886b32-8a1a-4cce-8729-dff55986dcca\") " Jul 08 18:28:17.784040 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.783987 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-bundle\") pod \"52021da0-e11d-41eb-846a-f1074a6639c4\" (UID: \"52021da0-e11d-41eb-846a-f1074a6639c4\") " Jul 08 18:28:17.784527 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.784500 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-bundle" (OuterVolumeSpecName: "bundle") pod "52021da0-e11d-41eb-846a-f1074a6639c4" (UID: "52021da0-e11d-41eb-846a-f1074a6639c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:17.784616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.784506 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-bundle" (OuterVolumeSpecName: "bundle") pod "57886b32-8a1a-4cce-8729-dff55986dcca" (UID: "57886b32-8a1a-4cce-8729-dff55986dcca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:17.785997 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.785975 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52021da0-e11d-41eb-846a-f1074a6639c4-kube-api-access-2gsq5" (OuterVolumeSpecName: "kube-api-access-2gsq5") pod "52021da0-e11d-41eb-846a-f1074a6639c4" (UID: "52021da0-e11d-41eb-846a-f1074a6639c4"). InnerVolumeSpecName "kube-api-access-2gsq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:28:17.786080 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.786048 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57886b32-8a1a-4cce-8729-dff55986dcca-kube-api-access-45xrw" (OuterVolumeSpecName: "kube-api-access-45xrw") pod "57886b32-8a1a-4cce-8729-dff55986dcca" (UID: "57886b32-8a1a-4cce-8729-dff55986dcca"). InnerVolumeSpecName "kube-api-access-45xrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:28:17.792233 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.792183 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-util" (OuterVolumeSpecName: "util") pod "52021da0-e11d-41eb-846a-f1074a6639c4" (UID: "52021da0-e11d-41eb-846a-f1074a6639c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:17.837104 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.837079 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-util" (OuterVolumeSpecName: "util") pod "57886b32-8a1a-4cce-8729-dff55986dcca" (UID: "57886b32-8a1a-4cce-8729-dff55986dcca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:17.885197 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.885178 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-util\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:17.885197 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.885197 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-bundle\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:17.885317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.885206 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gsq5\" (UniqueName: \"kubernetes.io/projected/52021da0-e11d-41eb-846a-f1074a6639c4-kube-api-access-2gsq5\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:17.885317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.885216 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45xrw\" (UniqueName: \"kubernetes.io/projected/57886b32-8a1a-4cce-8729-dff55986dcca-kube-api-access-45xrw\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:17.885317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.885225 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52021da0-e11d-41eb-846a-f1074a6639c4-bundle\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:17.885317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:17.885232 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57886b32-8a1a-4cce-8729-dff55986dcca-util\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:18.604846 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.604819 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" Jul 08 18:28:18.605293 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.604845 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp" event={"ID":"52021da0-e11d-41eb-846a-f1074a6639c4","Type":"ContainerDied","Data":"3e0b8de673682e7fdc3624d4da18928b84547ad2b67206db11580851f0cd6460"} Jul 08 18:28:18.605293 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.604896 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e0b8de673682e7fdc3624d4da18928b84547ad2b67206db11580851f0cd6460" Jul 08 18:28:18.606912 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.606887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" event={"ID":"57886b32-8a1a-4cce-8729-dff55986dcca","Type":"ContainerDied","Data":"d62a33f3a1d44a57abde2fcbdcc28edd7f4c9daf6abac56c64bd46bab1a292ec"} Jul 08 18:28:18.607089 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.606926 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d62a33f3a1d44a57abde2fcbdcc28edd7f4c9daf6abac56c64bd46bab1a292ec" Jul 08 18:28:18.607089 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.607047 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p" Jul 08 18:28:18.733402 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.733380 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:18.791972 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.791946 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-bundle\") pod \"2f2007a1-b0d4-4579-9875-c29531b12604\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " Jul 08 18:28:18.792109 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.792020 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcgkl\" (UniqueName: \"kubernetes.io/projected/2f2007a1-b0d4-4579-9875-c29531b12604-kube-api-access-tcgkl\") pod \"2f2007a1-b0d4-4579-9875-c29531b12604\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " Jul 08 18:28:18.792109 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.792071 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-util\") pod \"2f2007a1-b0d4-4579-9875-c29531b12604\" (UID: \"2f2007a1-b0d4-4579-9875-c29531b12604\") " Jul 08 18:28:18.792625 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.792601 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-bundle" (OuterVolumeSpecName: "bundle") pod "2f2007a1-b0d4-4579-9875-c29531b12604" (UID: "2f2007a1-b0d4-4579-9875-c29531b12604"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:18.794224 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.794203 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2007a1-b0d4-4579-9875-c29531b12604-kube-api-access-tcgkl" (OuterVolumeSpecName: "kube-api-access-tcgkl") pod "2f2007a1-b0d4-4579-9875-c29531b12604" (UID: "2f2007a1-b0d4-4579-9875-c29531b12604"). InnerVolumeSpecName "kube-api-access-tcgkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:28:18.861915 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.861865 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-util" (OuterVolumeSpecName: "util") pod "2f2007a1-b0d4-4579-9875-c29531b12604" (UID: "2f2007a1-b0d4-4579-9875-c29531b12604"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:18.893105 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.893077 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcgkl\" (UniqueName: \"kubernetes.io/projected/2f2007a1-b0d4-4579-9875-c29531b12604-kube-api-access-tcgkl\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:18.893105 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.893102 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-util\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:18.893244 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:18.893111 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2007a1-b0d4-4579-9875-c29531b12604-bundle\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:19.614355 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:19.614326 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" Jul 08 18:28:19.614355 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:19.614332 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j" event={"ID":"2f2007a1-b0d4-4579-9875-c29531b12604","Type":"ContainerDied","Data":"d9315bea9e817b881caf8c3487c9ba2c18f3d34764ee0564213530338cea319e"} Jul 08 18:28:19.614783 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:19.614376 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9315bea9e817b881caf8c3487c9ba2c18f3d34764ee0564213530338cea319e" Jul 08 18:28:29.771925 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.771886 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v"] Jul 08 18:28:29.772411 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772394 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerName="extract" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772414 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerName="extract" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772424 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57886b32-8a1a-4cce-8729-dff55986dcca" containerName="extract" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772430 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57886b32-8a1a-4cce-8729-dff55986dcca" containerName="extract" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772436 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57886b32-8a1a-4cce-8729-dff55986dcca" containerName="util" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772442 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57886b32-8a1a-4cce-8729-dff55986dcca" containerName="util" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772448 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52021da0-e11d-41eb-846a-f1074a6639c4" containerName="extract" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772453 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52021da0-e11d-41eb-846a-f1074a6639c4" containerName="extract" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772460 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerName="pull" Jul 08 18:28:29.772467 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772465 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerName="pull" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772479 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2007a1-b0d4-4579-9875-c29531b12604" containerName="util" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772485 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2007a1-b0d4-4579-9875-c29531b12604" containerName="util" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772492 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52021da0-e11d-41eb-846a-f1074a6639c4" containerName="util" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772497 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52021da0-e11d-41eb-846a-f1074a6639c4" containerName="util" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772504 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57886b32-8a1a-4cce-8729-dff55986dcca" containerName="pull" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772509 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57886b32-8a1a-4cce-8729-dff55986dcca" containerName="pull" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772519 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2007a1-b0d4-4579-9875-c29531b12604" containerName="pull" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772523 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2007a1-b0d4-4579-9875-c29531b12604" containerName="pull" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772536 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerName="util" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772540 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerName="util" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772546 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52021da0-e11d-41eb-846a-f1074a6639c4" containerName="pull" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772551 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52021da0-e11d-41eb-846a-f1074a6639c4" containerName="pull" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772558 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2007a1-b0d4-4579-9875-c29531b12604" containerName="extract" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772563 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2007a1-b0d4-4579-9875-c29531b12604" containerName="extract" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772618 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2007a1-b0d4-4579-9875-c29531b12604" containerName="extract" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772627 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52021da0-e11d-41eb-846a-f1074a6639c4" containerName="extract" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772634 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57886b32-8a1a-4cce-8729-dff55986dcca" containerName="extract" Jul 08 18:28:29.772771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.772641 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db70365-7b9e-4e32-9c87-05bf0c316ccb" containerName="extract" Jul 08 18:28:29.780068 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.780049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:29.782642 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.782621 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gc8xq\"" Jul 08 18:28:29.787471 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.787447 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v"] Jul 08 18:28:29.877090 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.877066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwz54\" (UniqueName: \"kubernetes.io/projected/b09bb296-22c5-4207-a149-06a87111987b-kube-api-access-dwz54\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" (UID: \"b09bb296-22c5-4207-a149-06a87111987b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:29.877193 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.877100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b09bb296-22c5-4207-a149-06a87111987b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" (UID: \"b09bb296-22c5-4207-a149-06a87111987b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:29.977878 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.977853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwz54\" (UniqueName: \"kubernetes.io/projected/b09bb296-22c5-4207-a149-06a87111987b-kube-api-access-dwz54\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" (UID: \"b09bb296-22c5-4207-a149-06a87111987b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:29.977964 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.977888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b09bb296-22c5-4207-a149-06a87111987b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" (UID: \"b09bb296-22c5-4207-a149-06a87111987b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:29.978202 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.978187 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b09bb296-22c5-4207-a149-06a87111987b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" (UID: \"b09bb296-22c5-4207-a149-06a87111987b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:29.989131 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:29.989106 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwz54\" (UniqueName: \"kubernetes.io/projected/b09bb296-22c5-4207-a149-06a87111987b-kube-api-access-dwz54\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" (UID: \"b09bb296-22c5-4207-a149-06a87111987b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:30.091641 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:30.091588 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:30.229472 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:30.229445 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v"] Jul 08 18:28:30.230726 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:30.230703 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09bb296_22c5_4207_a149_06a87111987b.slice/crio-b8c9f7e0161c946fd93e04515c45d27dacd3dabf83a0696490e5746fbf244812 WatchSource:0}: Error finding container b8c9f7e0161c946fd93e04515c45d27dacd3dabf83a0696490e5746fbf244812: Status 404 returned error can't find the container with id b8c9f7e0161c946fd93e04515c45d27dacd3dabf83a0696490e5746fbf244812 Jul 08 18:28:30.678040 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:30.678000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" event={"ID":"b09bb296-22c5-4207-a149-06a87111987b","Type":"ContainerStarted","Data":"b8c9f7e0161c946fd93e04515c45d27dacd3dabf83a0696490e5746fbf244812"} Jul 08 18:28:31.718167 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:31.718137 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2"] Jul 08 18:28:31.721736 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:31.721721 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:31.724393 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:31.724372 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kwdsk\"" Jul 08 18:28:31.730815 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:31.730791 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2"] Jul 08 18:28:31.791683 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:31.791647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drzxg\" (UniqueName: \"kubernetes.io/projected/aad87807-844f-4322-b3d5-25d355bc1bba-kube-api-access-drzxg\") pod \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" (UID: \"aad87807-844f-4322-b3d5-25d355bc1bba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:31.892548 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:31.892519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drzxg\" (UniqueName: \"kubernetes.io/projected/aad87807-844f-4322-b3d5-25d355bc1bba-kube-api-access-drzxg\") pod \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" (UID: \"aad87807-844f-4322-b3d5-25d355bc1bba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:31.901001 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:31.900972 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drzxg\" (UniqueName: \"kubernetes.io/projected/aad87807-844f-4322-b3d5-25d355bc1bba-kube-api-access-drzxg\") pod \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" (UID: \"aad87807-844f-4322-b3d5-25d355bc1bba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:32.032803 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:32.032780 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:32.187823 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:32.187787 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2"] Jul 08 18:28:32.188222 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:32.188194 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad87807_844f_4322_b3d5_25d355bc1bba.slice/crio-0ee8b1e65e42ca453ed5c7a39d994ee8a8db48eda6ed0f4a8c25cf6f95c84010 WatchSource:0}: Error finding container 0ee8b1e65e42ca453ed5c7a39d994ee8a8db48eda6ed0f4a8c25cf6f95c84010: Status 404 returned error can't find the container with id 0ee8b1e65e42ca453ed5c7a39d994ee8a8db48eda6ed0f4a8c25cf6f95c84010 Jul 08 18:28:32.690173 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:32.690141 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" event={"ID":"aad87807-844f-4322-b3d5-25d355bc1bba","Type":"ContainerStarted","Data":"0ee8b1e65e42ca453ed5c7a39d994ee8a8db48eda6ed0f4a8c25cf6f95c84010"} Jul 08 18:28:34.227117 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.227023 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2"] Jul 08 18:28:34.230753 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.230734 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" Jul 08 18:28:34.233293 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.233269 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lpt48\"" Jul 08 18:28:34.240257 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.240237 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2"] Jul 08 18:28:34.317884 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.317845 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddld\" (UniqueName: \"kubernetes.io/projected/266baead-51c9-445c-9a27-292512147c66-kube-api-access-vddld\") pod \"authorino-operator-75cc8d5cb7-bqmq2\" (UID: \"266baead-51c9-445c-9a27-292512147c66\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" Jul 08 18:28:34.418766 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.418732 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddld\" (UniqueName: \"kubernetes.io/projected/266baead-51c9-445c-9a27-292512147c66-kube-api-access-vddld\") pod \"authorino-operator-75cc8d5cb7-bqmq2\" (UID: \"266baead-51c9-445c-9a27-292512147c66\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" Jul 08 18:28:34.428339 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.428314 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddld\" (UniqueName: \"kubernetes.io/projected/266baead-51c9-445c-9a27-292512147c66-kube-api-access-vddld\") pod \"authorino-operator-75cc8d5cb7-bqmq2\" (UID: \"266baead-51c9-445c-9a27-292512147c66\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" Jul 08 18:28:34.541324 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.541293 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" Jul 08 18:28:34.707055 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.707020 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" event={"ID":"aad87807-844f-4322-b3d5-25d355bc1bba","Type":"ContainerStarted","Data":"a34fcac2ca7a6998575ff28b848572325b2e9c9de0443d1d746bb5958d8291f2"} Jul 08 18:28:34.707313 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.707089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:34.731487 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:34.731433 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" podStartSLOduration=2.0167313 podStartE2EDuration="3.731416024s" totalImagesPullingTime="1.714684724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:34.729055917 +0000 UTC m=+649.352643022" watchObservedRunningTime="2026-07-08 18:28:34.731416024 +0000 UTC m=+649.355003105" Jul 08 18:28:35.620351 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:35.620325 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2"] Jul 08 18:28:35.622317 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:35.622285 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266baead_51c9_445c_9a27_292512147c66.slice/crio-fe51a3892082f6d43b94cf53ea3853aecc445159a27df7d648a9aff2f2efd944 WatchSource:0}: Error finding container fe51a3892082f6d43b94cf53ea3853aecc445159a27df7d648a9aff2f2efd944: Status 404 returned error can't find the container with id fe51a3892082f6d43b94cf53ea3853aecc445159a27df7d648a9aff2f2efd944 Jul 08 18:28:35.713881 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:35.713852 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" event={"ID":"b09bb296-22c5-4207-a149-06a87111987b","Type":"ContainerStarted","Data":"61068bddec1b0bf9c56fb4c95a75b2cdffb83e5d80e56bdeec72b0c391688db9"} Jul 08 18:28:35.713991 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:35.713967 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:35.715242 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:35.715212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" event={"ID":"266baead-51c9-445c-9a27-292512147c66","Type":"ContainerStarted","Data":"fe51a3892082f6d43b94cf53ea3853aecc445159a27df7d648a9aff2f2efd944"} Jul 08 18:28:35.741926 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:35.741850 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" podStartSLOduration=1.428577142 podStartE2EDuration="6.741838076s" totalImagesPullingTime="5.313260934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:35.739957996 +0000 UTC m=+650.363545075" watchObservedRunningTime="2026-07-08 18:28:35.741838076 +0000 UTC m=+650.365425156" Jul 08 18:28:37.629469 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.629442 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms"] Jul 08 18:28:37.633581 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.633526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" Jul 08 18:28:37.636095 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.636074 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 18:28:37.636095 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.636084 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gx5tv\"" Jul 08 18:28:37.644049 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.644026 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms"] Jul 08 18:28:37.747753 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.747722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chgx\" (UniqueName: \"kubernetes.io/projected/b1fcac96-320f-407d-9461-b007c5d314fd-kube-api-access-2chgx\") pod \"dns-operator-controller-manager-864dd78c4c-s69ms\" (UID: \"b1fcac96-320f-407d-9461-b007c5d314fd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" Jul 08 18:28:37.848737 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.848717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chgx\" (UniqueName: \"kubernetes.io/projected/b1fcac96-320f-407d-9461-b007c5d314fd-kube-api-access-2chgx\") pod \"dns-operator-controller-manager-864dd78c4c-s69ms\" (UID: \"b1fcac96-320f-407d-9461-b007c5d314fd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" Jul 08 18:28:37.856742 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.856713 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chgx\" (UniqueName: \"kubernetes.io/projected/b1fcac96-320f-407d-9461-b007c5d314fd-kube-api-access-2chgx\") pod \"dns-operator-controller-manager-864dd78c4c-s69ms\" (UID: \"b1fcac96-320f-407d-9461-b007c5d314fd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" Jul 08 18:28:37.946927 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:37.946905 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" Jul 08 18:28:38.079164 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:38.079138 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms"] Jul 08 18:28:38.080825 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:38.080797 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fcac96_320f_407d_9461_b007c5d314fd.slice/crio-f4cea21b10440855c4944d451ca054cff1e782a8b1b179d0bad69af521a3f235 WatchSource:0}: Error finding container f4cea21b10440855c4944d451ca054cff1e782a8b1b179d0bad69af521a3f235: Status 404 returned error can't find the container with id f4cea21b10440855c4944d451ca054cff1e782a8b1b179d0bad69af521a3f235 Jul 08 18:28:38.736260 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:38.736229 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" event={"ID":"266baead-51c9-445c-9a27-292512147c66","Type":"ContainerStarted","Data":"3de6cf111e46e31d8bdb1cd558f6be0a758deb5b80059ebd2a8a66b714ac8fca"} Jul 08 18:28:38.736649 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:38.736420 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" Jul 08 18:28:38.737600 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:38.737579 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" event={"ID":"b1fcac96-320f-407d-9461-b007c5d314fd","Type":"ContainerStarted","Data":"f4cea21b10440855c4944d451ca054cff1e782a8b1b179d0bad69af521a3f235"} Jul 08 18:28:38.759280 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:38.759230 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" podStartSLOduration=2.542826399 podStartE2EDuration="4.759212993s" totalImagesPullingTime="2.216386594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:38.755445426 +0000 UTC m=+653.379032507" watchObservedRunningTime="2026-07-08 18:28:38.759212993 +0000 UTC m=+653.382800075" Jul 08 18:28:40.750161 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:40.750129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" event={"ID":"b1fcac96-320f-407d-9461-b007c5d314fd","Type":"ContainerStarted","Data":"0eee85351e35ba429773a83c73c1838218b38eeeed0da3d414b802bc4944126c"} Jul 08 18:28:40.750526 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:40.750219 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" Jul 08 18:28:40.767546 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:40.767506 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" podStartSLOduration=1.6406024700000001 podStartE2EDuration="3.767492933s" totalImagesPullingTime="2.126890463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:40.766830036 +0000 UTC m=+655.390417135" watchObservedRunningTime="2026-07-08 18:28:40.767492933 +0000 UTC m=+655.391080013" Jul 08 18:28:45.717253 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:45.717222 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:46.723679 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:46.723642 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:48.551312 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.551279 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v"] Jul 08 18:28:48.551783 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.551472 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" podUID="b09bb296-22c5-4207-a149-06a87111987b" containerName="manager" containerID="cri-o://61068bddec1b0bf9c56fb4c95a75b2cdffb83e5d80e56bdeec72b0c391688db9" gracePeriod=2 Jul 08 18:28:48.557141 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.557105 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v"] Jul 08 18:28:48.576299 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.575931 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2"] Jul 08 18:28:48.576299 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.576147 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" containerName="manager" containerID="cri-o://a34fcac2ca7a6998575ff28b848572325b2e9c9de0443d1d746bb5958d8291f2" gracePeriod=2 Jul 08 18:28:48.578690 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.578564 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:48.578844 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.578827 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d"] Jul 08 18:28:48.579522 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.579500 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09bb296-22c5-4207-a149-06a87111987b" containerName="manager" Jul 08 18:28:48.579603 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.579526 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09bb296-22c5-4207-a149-06a87111987b" containerName="manager" Jul 08 18:28:48.579655 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.579644 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b09bb296-22c5-4207-a149-06a87111987b" containerName="manager" Jul 08 18:28:48.582862 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.582845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:48.591458 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.591436 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2"] Jul 08 18:28:48.597275 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.597214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d"] Jul 08 18:28:48.602994 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.602955 2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"aad87807-844f-4322-b3d5-25d355bc1bba\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-5xjb2\": pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:48.607113 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.607090 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4"] Jul 08 18:28:48.607484 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.607470 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" containerName="manager" Jul 08 18:28:48.607484 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.607485 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" containerName="manager" Jul 08 18:28:48.607583 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.607573 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" containerName="manager" Jul 08 18:28:48.610607 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.610586 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" Jul 08 18:28:48.624699 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.624673 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4"] Jul 08 18:28:48.624948 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.624921 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:48.628266 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.628233 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:48.630603 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.630581 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:48.632697 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.632650 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:48.738017 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.737988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9gnf\" (UniqueName: \"kubernetes.io/projected/ac0c603e-0b25-4df0-a0f0-3588dcb74359-kube-api-access-n9gnf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kn42d\" (UID: \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:48.738280 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.738259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0c603e-0b25-4df0-a0f0-3588dcb74359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kn42d\" (UID: \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:48.738391 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.738369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kh99\" (UniqueName: \"kubernetes.io/projected/d72084c8-cce0-4f8f-8de5-868e03bf2900-kube-api-access-9kh99\") pod \"limitador-operator-controller-manager-859d7b78bd-557r4\" (UID: \"d72084c8-cce0-4f8f-8de5-868e03bf2900\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" Jul 08 18:28:48.798397 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.798365 2565 generic.go:356] "Generic (PLEG): container finished" podID="b09bb296-22c5-4207-a149-06a87111987b" containerID="61068bddec1b0bf9c56fb4c95a75b2cdffb83e5d80e56bdeec72b0c391688db9" exitCode=0 Jul 08 18:28:48.808031 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.808002 2565 generic.go:356] "Generic (PLEG): container finished" podID="aad87807-844f-4322-b3d5-25d355bc1bba" containerID="a34fcac2ca7a6998575ff28b848572325b2e9c9de0443d1d746bb5958d8291f2" exitCode=0 Jul 08 18:28:48.827201 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.827183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:48.829949 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.829928 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:48.830073 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.830027 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:48.832699 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.832655 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:48.835065 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.835043 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:48.837398 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:48.837379 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:48.839807 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.839790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0c603e-0b25-4df0-a0f0-3588dcb74359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kn42d\" (UID: \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:48.839865 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.839832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kh99\" (UniqueName: \"kubernetes.io/projected/d72084c8-cce0-4f8f-8de5-868e03bf2900-kube-api-access-9kh99\") pod \"limitador-operator-controller-manager-859d7b78bd-557r4\" (UID: \"d72084c8-cce0-4f8f-8de5-868e03bf2900\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" Jul 08 18:28:48.839975 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.839955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9gnf\" (UniqueName: \"kubernetes.io/projected/ac0c603e-0b25-4df0-a0f0-3588dcb74359-kube-api-access-n9gnf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kn42d\" (UID: \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:48.840147 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.840130 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0c603e-0b25-4df0-a0f0-3588dcb74359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kn42d\" (UID: \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:48.855524 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.855505 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kh99\" (UniqueName: \"kubernetes.io/projected/d72084c8-cce0-4f8f-8de5-868e03bf2900-kube-api-access-9kh99\") pod \"limitador-operator-controller-manager-859d7b78bd-557r4\" (UID: \"d72084c8-cce0-4f8f-8de5-868e03bf2900\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" Jul 08 18:28:48.855604 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.855523 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9gnf\" (UniqueName: \"kubernetes.io/projected/ac0c603e-0b25-4df0-a0f0-3588dcb74359-kube-api-access-n9gnf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kn42d\" (UID: \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:48.940278 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.940255 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwz54\" (UniqueName: \"kubernetes.io/projected/b09bb296-22c5-4207-a149-06a87111987b-kube-api-access-dwz54\") pod \"b09bb296-22c5-4207-a149-06a87111987b\" (UID: \"b09bb296-22c5-4207-a149-06a87111987b\") " Jul 08 18:28:48.940388 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.940289 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b09bb296-22c5-4207-a149-06a87111987b-extensions-socket-volume\") pod \"b09bb296-22c5-4207-a149-06a87111987b\" (UID: \"b09bb296-22c5-4207-a149-06a87111987b\") " Jul 08 18:28:48.940388 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.940314 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drzxg\" (UniqueName: \"kubernetes.io/projected/aad87807-844f-4322-b3d5-25d355bc1bba-kube-api-access-drzxg\") pod \"aad87807-844f-4322-b3d5-25d355bc1bba\" (UID: \"aad87807-844f-4322-b3d5-25d355bc1bba\") " Jul 08 18:28:48.940686 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.940644 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09bb296-22c5-4207-a149-06a87111987b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b09bb296-22c5-4207-a149-06a87111987b" (UID: "b09bb296-22c5-4207-a149-06a87111987b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:28:48.942174 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.942151 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09bb296-22c5-4207-a149-06a87111987b-kube-api-access-dwz54" (OuterVolumeSpecName: "kube-api-access-dwz54") pod "b09bb296-22c5-4207-a149-06a87111987b" (UID: "b09bb296-22c5-4207-a149-06a87111987b"). InnerVolumeSpecName "kube-api-access-dwz54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:28:48.942238 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:48.942170 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad87807-844f-4322-b3d5-25d355bc1bba-kube-api-access-drzxg" (OuterVolumeSpecName: "kube-api-access-drzxg") pod "aad87807-844f-4322-b3d5-25d355bc1bba" (UID: "aad87807-844f-4322-b3d5-25d355bc1bba"). InnerVolumeSpecName "kube-api-access-drzxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:28:49.000351 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.000331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:49.007270 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.007255 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" Jul 08 18:28:49.041566 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.041534 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwz54\" (UniqueName: \"kubernetes.io/projected/b09bb296-22c5-4207-a149-06a87111987b-kube-api-access-dwz54\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:49.041566 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.041565 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b09bb296-22c5-4207-a149-06a87111987b-extensions-socket-volume\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:49.041767 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.041578 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drzxg\" (UniqueName: \"kubernetes.io/projected/aad87807-844f-4322-b3d5-25d355bc1bba-kube-api-access-drzxg\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:28:49.149001 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.148979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d"] Jul 08 18:28:49.151678 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:49.151640 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0c603e_0b25_4df0_a0f0_3588dcb74359.slice/crio-faaf792f8976a2522b88cabb7ec33e8d3c3ec1177fb19ac69e8d9a3b9dc46329 WatchSource:0}: Error finding container faaf792f8976a2522b88cabb7ec33e8d3c3ec1177fb19ac69e8d9a3b9dc46329: Status 404 returned error can't find the container with id faaf792f8976a2522b88cabb7ec33e8d3c3ec1177fb19ac69e8d9a3b9dc46329 Jul 08 18:28:49.166580 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.166543 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4"] Jul 08 18:28:49.168627 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:49.168601 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72084c8_cce0_4f8f_8de5_868e03bf2900.slice/crio-f7b3d587d326d67b4af3bbcf46b84b840a6a997427d869695d9c287e8f8e1e74 WatchSource:0}: Error finding container f7b3d587d326d67b4af3bbcf46b84b840a6a997427d869695d9c287e8f8e1e74: Status 404 returned error can't find the container with id f7b3d587d326d67b4af3bbcf46b84b840a6a997427d869695d9c287e8f8e1e74 Jul 08 18:28:49.708765 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.708737 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw"] Jul 08 18:28:49.712913 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.712882 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:49.726176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.726151 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw"] Jul 08 18:28:49.745570 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.745548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bqmq2" Jul 08 18:28:49.746445 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.746422 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:49.748282 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.748256 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:49.750231 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.750210 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:49.752110 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.752088 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:49.817138 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.817112 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:49.817299 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.817252 2565 scope.go:117] "RemoveContainer" containerID="61068bddec1b0bf9c56fb4c95a75b2cdffb83e5d80e56bdeec72b0c391688db9" Jul 08 18:28:49.819244 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.819224 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:49.819587 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.819561 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:49.821850 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.821821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" event={"ID":"d72084c8-cce0-4f8f-8de5-868e03bf2900","Type":"ContainerStarted","Data":"d3dbac57d08aa6c8143b81aa6680c151abf4977ef9fba4f705bade7c183f6261"} Jul 08 18:28:49.821946 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.821874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" event={"ID":"d72084c8-cce0-4f8f-8de5-868e03bf2900","Type":"ContainerStarted","Data":"f7b3d587d326d67b4af3bbcf46b84b840a6a997427d869695d9c287e8f8e1e74"} Jul 08 18:28:49.822610 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.822268 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:49.822610 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.822322 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" Jul 08 18:28:49.827055 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.827030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" event={"ID":"ac0c603e-0b25-4df0-a0f0-3588dcb74359","Type":"ContainerStarted","Data":"18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681"} Jul 08 18:28:49.827159 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.827079 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" event={"ID":"ac0c603e-0b25-4df0-a0f0-3588dcb74359","Type":"ContainerStarted","Data":"faaf792f8976a2522b88cabb7ec33e8d3c3ec1177fb19ac69e8d9a3b9dc46329"} Jul 08 18:28:49.827159 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.827123 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:28:49.829311 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.829294 2565 scope.go:117] "RemoveContainer" containerID="a34fcac2ca7a6998575ff28b848572325b2e9c9de0443d1d746bb5958d8291f2" Jul 08 18:28:49.845212 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.845175 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" podStartSLOduration=1.845163627 podStartE2EDuration="1.845163627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:49.844030103 +0000 UTC m=+664.467617183" watchObservedRunningTime="2026-07-08 18:28:49.845163627 +0000 UTC m=+664.468750708" Jul 08 18:28:49.846119 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.846096 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:49.847161 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.847139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9ncb\" (UniqueName: \"kubernetes.io/projected/26d52641-106b-48ec-9244-e8995c454aec-kube-api-access-r9ncb\") pod \"kuadrant-operator-controller-manager-86878c769f-76mzw\" (UID: \"26d52641-106b-48ec-9244-e8995c454aec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:49.847265 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.847168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26d52641-106b-48ec-9244-e8995c454aec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76mzw\" (UID: \"26d52641-106b-48ec-9244-e8995c454aec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:49.848019 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.847998 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:49.849806 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.849787 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpr6v\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="b09bb296-22c5-4207-a149-06a87111987b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpr6v" Jul 08 18:28:49.851671 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:28:49.851637 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5xjb2\" is forbidden: User \"system:node:ip-10-0-136-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-178.ec2.internal' and this object" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5xjb2" Jul 08 18:28:49.875803 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.875769 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" podStartSLOduration=1.8757579899999999 podStartE2EDuration="1.87575799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:49.873382254 +0000 UTC m=+664.496969335" watchObservedRunningTime="2026-07-08 18:28:49.87575799 +0000 UTC m=+664.499345116" Jul 08 18:28:49.928288 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.928255 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad87807-844f-4322-b3d5-25d355bc1bba" path="/var/lib/kubelet/pods/aad87807-844f-4322-b3d5-25d355bc1bba/volumes" Jul 08 18:28:49.928678 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.928649 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b09bb296-22c5-4207-a149-06a87111987b" path="/var/lib/kubelet/pods/b09bb296-22c5-4207-a149-06a87111987b/volumes" Jul 08 18:28:49.948000 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.947981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9ncb\" (UniqueName: \"kubernetes.io/projected/26d52641-106b-48ec-9244-e8995c454aec-kube-api-access-r9ncb\") pod \"kuadrant-operator-controller-manager-86878c769f-76mzw\" (UID: \"26d52641-106b-48ec-9244-e8995c454aec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:49.948095 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.948015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26d52641-106b-48ec-9244-e8995c454aec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76mzw\" (UID: \"26d52641-106b-48ec-9244-e8995c454aec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:49.948347 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.948327 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26d52641-106b-48ec-9244-e8995c454aec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76mzw\" (UID: \"26d52641-106b-48ec-9244-e8995c454aec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:49.958196 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:49.958173 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9ncb\" (UniqueName: \"kubernetes.io/projected/26d52641-106b-48ec-9244-e8995c454aec-kube-api-access-r9ncb\") pod \"kuadrant-operator-controller-manager-86878c769f-76mzw\" (UID: \"26d52641-106b-48ec-9244-e8995c454aec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:50.023822 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:50.023792 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:50.160717 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:50.160693 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw"] Jul 08 18:28:50.162295 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:28:50.162271 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d52641_106b_48ec_9244_e8995c454aec.slice/crio-1e0e4a76f36a4a8128e8e4031027a9c45a75966b031725728e6af22b70cd44ae WatchSource:0}: Error finding container 1e0e4a76f36a4a8128e8e4031027a9c45a75966b031725728e6af22b70cd44ae: Status 404 returned error can't find the container with id 1e0e4a76f36a4a8128e8e4031027a9c45a75966b031725728e6af22b70cd44ae Jul 08 18:28:50.834534 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:50.834503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" event={"ID":"26d52641-106b-48ec-9244-e8995c454aec","Type":"ContainerStarted","Data":"1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025"} Jul 08 18:28:50.835021 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:50.834549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" event={"ID":"26d52641-106b-48ec-9244-e8995c454aec","Type":"ContainerStarted","Data":"1e0e4a76f36a4a8128e8e4031027a9c45a75966b031725728e6af22b70cd44ae"} Jul 08 18:28:50.835021 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:50.834572 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:28:50.854086 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:50.854044 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" podStartSLOduration=1.854029044 podStartE2EDuration="1.854029044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:28:50.852885753 +0000 UTC m=+665.476472857" watchObservedRunningTime="2026-07-08 18:28:50.854029044 +0000 UTC m=+665.477616124" Jul 08 18:28:51.758171 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:28:51.758135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s69ms" Jul 08 18:29:00.838876 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:00.838848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:29:01.843629 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:01.843599 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:29:01.844042 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:01.843774 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-557r4" Jul 08 18:29:01.904916 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:01.904881 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d"] Jul 08 18:29:01.905091 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:01.905026 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" podUID="ac0c603e-0b25-4df0-a0f0-3588dcb74359" containerName="manager" containerID="cri-o://18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681" gracePeriod=10 Jul 08 18:29:02.148283 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.148258 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:29:02.244406 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.244378 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9gnf\" (UniqueName: \"kubernetes.io/projected/ac0c603e-0b25-4df0-a0f0-3588dcb74359-kube-api-access-n9gnf\") pod \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\" (UID: \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\") " Jul 08 18:29:02.244539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.244452 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0c603e-0b25-4df0-a0f0-3588dcb74359-extensions-socket-volume\") pod \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\" (UID: \"ac0c603e-0b25-4df0-a0f0-3588dcb74359\") " Jul 08 18:29:02.244822 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.244800 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0c603e-0b25-4df0-a0f0-3588dcb74359-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ac0c603e-0b25-4df0-a0f0-3588dcb74359" (UID: "ac0c603e-0b25-4df0-a0f0-3588dcb74359"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:29:02.246270 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.246252 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0c603e-0b25-4df0-a0f0-3588dcb74359-kube-api-access-n9gnf" (OuterVolumeSpecName: "kube-api-access-n9gnf") pod "ac0c603e-0b25-4df0-a0f0-3588dcb74359" (UID: "ac0c603e-0b25-4df0-a0f0-3588dcb74359"). InnerVolumeSpecName "kube-api-access-n9gnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:29:02.345498 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.345476 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9gnf\" (UniqueName: \"kubernetes.io/projected/ac0c603e-0b25-4df0-a0f0-3588dcb74359-kube-api-access-n9gnf\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:29:02.345498 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.345496 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0c603e-0b25-4df0-a0f0-3588dcb74359-extensions-socket-volume\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:29:02.909456 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.909426 2565 generic.go:356] "Generic (PLEG): container finished" podID="ac0c603e-0b25-4df0-a0f0-3588dcb74359" containerID="18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681" exitCode=0 Jul 08 18:29:02.909977 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.909481 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" Jul 08 18:29:02.909977 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.909520 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" event={"ID":"ac0c603e-0b25-4df0-a0f0-3588dcb74359","Type":"ContainerDied","Data":"18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681"} Jul 08 18:29:02.909977 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.909603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d" event={"ID":"ac0c603e-0b25-4df0-a0f0-3588dcb74359","Type":"ContainerDied","Data":"faaf792f8976a2522b88cabb7ec33e8d3c3ec1177fb19ac69e8d9a3b9dc46329"} Jul 08 18:29:02.909977 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.909632 2565 scope.go:117] "RemoveContainer" containerID="18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681" Jul 08 18:29:02.921049 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.921032 2565 scope.go:117] "RemoveContainer" containerID="18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681" Jul 08 18:29:02.921306 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:29:02.921289 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681\": container with ID starting with 18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681 not found: ID does not exist" containerID="18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681" Jul 08 18:29:02.921366 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.921319 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681"} err="failed to get container status \"18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681\": rpc error: code = NotFound desc = could not find container \"18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681\": container with ID starting with 18c07839eb5c2fe281f85363c9e07e7c6747bc5dc16a027e2ac3a11c5a98a681 not found: ID does not exist" Jul 08 18:29:02.942703 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.942678 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d"] Jul 08 18:29:02.946509 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:02.946488 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kn42d"] Jul 08 18:29:03.927506 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:03.927475 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0c603e-0b25-4df0-a0f0-3588dcb74359" path="/var/lib/kubelet/pods/ac0c603e-0b25-4df0-a0f0-3588dcb74359/volumes" Jul 08 18:29:18.245734 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.245701 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4"] Jul 08 18:29:18.246317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.246302 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0c603e-0b25-4df0-a0f0-3588dcb74359" containerName="manager" Jul 08 18:29:18.246368 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.246320 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0c603e-0b25-4df0-a0f0-3588dcb74359" containerName="manager" Jul 08 18:29:18.246443 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.246432 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0c603e-0b25-4df0-a0f0-3588dcb74359" containerName="manager" Jul 08 18:29:18.248981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.248957 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.251643 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.251619 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 18:29:18.251812 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.251765 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vwb88\"" Jul 08 18:29:18.259815 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.259790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.259944 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.259860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7857d7c4-d98d-4e46-a602-8f7d5535baad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.260016 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.259981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.260187 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.260008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.260285 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.260199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.260285 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.260240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.260285 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.260263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.260446 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.260298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gjp\" (UniqueName: \"kubernetes.io/projected/7857d7c4-d98d-4e46-a602-8f7d5535baad-kube-api-access-22gjp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.260446 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.260333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.264053 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.264025 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4"] Jul 08 18:29:18.361019 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.360984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gjp\" (UniqueName: \"kubernetes.io/projected/7857d7c4-d98d-4e46-a602-8f7d5535baad-kube-api-access-22gjp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7857d7c4-d98d-4e46-a602-8f7d5535baad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361294 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361759 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361497 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361759 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361526 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361759 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361609 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.361759 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.361711 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.362187 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.362166 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7857d7c4-d98d-4e46-a602-8f7d5535baad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.363550 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.363522 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.363654 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.363641 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.375935 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.375909 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7857d7c4-d98d-4e46-a602-8f7d5535baad-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.377384 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.377360 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gjp\" (UniqueName: \"kubernetes.io/projected/7857d7c4-d98d-4e46-a602-8f7d5535baad-kube-api-access-22gjp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fbrl4\" (UID: \"7857d7c4-d98d-4e46-a602-8f7d5535baad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.560159 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.560083 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:18.701311 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.701289 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4"] Jul 08 18:29:18.702907 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:29:18.702880 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7857d7c4_d98d_4e46_a602_8f7d5535baad.slice/crio-fedce44f5a90afd23cb7c563140d103a245db08df81c1819308b2b69d0f49f10 WatchSource:0}: Error finding container fedce44f5a90afd23cb7c563140d103a245db08df81c1819308b2b69d0f49f10: Status 404 returned error can't find the container with id fedce44f5a90afd23cb7c563140d103a245db08df81c1819308b2b69d0f49f10 Jul 08 18:29:18.997966 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:18.997931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" event={"ID":"7857d7c4-d98d-4e46-a602-8f7d5535baad","Type":"ContainerStarted","Data":"fedce44f5a90afd23cb7c563140d103a245db08df81c1819308b2b69d0f49f10"} Jul 08 18:29:21.295470 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:21.295431 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:29:21.295782 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:21.295505 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:29:21.295782 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:21.295537 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:29:22.018963 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:22.018921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" event={"ID":"7857d7c4-d98d-4e46-a602-8f7d5535baad","Type":"ContainerStarted","Data":"d34626e64c824f00fe070fb96feeb7fe893fd80546b846cb7171cd7baeae4a33"} Jul 08 18:29:22.042176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:22.042133 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" podStartSLOduration=1.451774851 podStartE2EDuration="4.042119006s" totalImagesPullingTime="2.590344155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:29:22.039122245 +0000 UTC m=+696.662709324" watchObservedRunningTime="2026-07-08 18:29:22.042119006 +0000 UTC m=+696.665706085" Jul 08 18:29:22.560883 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:22.560857 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:22.565627 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:22.565608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:23.025504 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.025479 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:23.026368 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.026352 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:29:23.124906 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.124876 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nkzzp"] Jul 08 18:29:23.128021 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.128001 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:23.130608 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.130590 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 18:29:23.130838 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.130825 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kg9gr\"" Jul 08 18:29:23.137407 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.137385 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nkzzp"] Jul 08 18:29:23.204809 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.204781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-config-file\") pod \"limitador-limitador-64c94fd64f-nkzzp\" (UID: \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:23.205003 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.204879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc2v8\" (UniqueName: \"kubernetes.io/projected/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-kube-api-access-jc2v8\") pod \"limitador-limitador-64c94fd64f-nkzzp\" (UID: \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:23.224855 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.224828 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nkzzp"] Jul 08 18:29:23.306262 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.306188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc2v8\" (UniqueName: \"kubernetes.io/projected/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-kube-api-access-jc2v8\") pod \"limitador-limitador-64c94fd64f-nkzzp\" (UID: \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:23.306262 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.306258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-config-file\") pod \"limitador-limitador-64c94fd64f-nkzzp\" (UID: \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:23.306873 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.306855 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-config-file\") pod \"limitador-limitador-64c94fd64f-nkzzp\" (UID: \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:23.316400 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.316374 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc2v8\" (UniqueName: \"kubernetes.io/projected/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-kube-api-access-jc2v8\") pod \"limitador-limitador-64c94fd64f-nkzzp\" (UID: \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:23.439707 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.439653 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:23.582694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:23.582648 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nkzzp"] Jul 08 18:29:23.583793 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:29:23.583771 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d362f8b_a94d_4ce2_acb4_f5a42d6914a9.slice/crio-09c7e031bd1ba29c2195433709b9d834763298b12aeb97449e12a6347c645a17 WatchSource:0}: Error finding container 09c7e031bd1ba29c2195433709b9d834763298b12aeb97449e12a6347c645a17: Status 404 returned error can't find the container with id 09c7e031bd1ba29c2195433709b9d834763298b12aeb97449e12a6347c645a17 Jul 08 18:29:24.031523 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.031496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" event={"ID":"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9","Type":"ContainerStarted","Data":"09c7e031bd1ba29c2195433709b9d834763298b12aeb97449e12a6347c645a17"} Jul 08 18:29:24.137912 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.137882 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-87v27"] Jul 08 18:29:24.141002 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.140982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-87v27" Jul 08 18:29:24.143338 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.143316 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5ps7\"" Jul 08 18:29:24.151003 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.150972 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-87v27"] Jul 08 18:29:24.214860 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.214838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/8ec391ec-594e-43ad-b4a3-44ef948c2f8a-kube-api-access-q6ksr\") pod \"authorino-75d556bd77-87v27\" (UID: \"8ec391ec-594e-43ad-b4a3-44ef948c2f8a\") " pod="kuadrant-system/authorino-75d556bd77-87v27" Jul 08 18:29:24.315335 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.315276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/8ec391ec-594e-43ad-b4a3-44ef948c2f8a-kube-api-access-q6ksr\") pod \"authorino-75d556bd77-87v27\" (UID: \"8ec391ec-594e-43ad-b4a3-44ef948c2f8a\") " pod="kuadrant-system/authorino-75d556bd77-87v27" Jul 08 18:29:24.323114 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.323084 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/8ec391ec-594e-43ad-b4a3-44ef948c2f8a-kube-api-access-q6ksr\") pod \"authorino-75d556bd77-87v27\" (UID: \"8ec391ec-594e-43ad-b4a3-44ef948c2f8a\") " pod="kuadrant-system/authorino-75d556bd77-87v27" Jul 08 18:29:24.452824 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.452801 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-87v27" Jul 08 18:29:24.634956 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:24.634922 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-87v27"] Jul 08 18:29:24.635888 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:29:24.635857 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ec391ec_594e_43ad_b4a3_44ef948c2f8a.slice/crio-2e997cccee8c7a460deaa25fb8f0509a781dbce7e785e3060437eb3c28f9e14f WatchSource:0}: Error finding container 2e997cccee8c7a460deaa25fb8f0509a781dbce7e785e3060437eb3c28f9e14f: Status 404 returned error can't find the container with id 2e997cccee8c7a460deaa25fb8f0509a781dbce7e785e3060437eb3c28f9e14f Jul 08 18:29:25.040234 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:25.040198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-87v27" event={"ID":"8ec391ec-594e-43ad-b4a3-44ef948c2f8a","Type":"ContainerStarted","Data":"2e997cccee8c7a460deaa25fb8f0509a781dbce7e785e3060437eb3c28f9e14f"} Jul 08 18:29:29.068907 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:29.068873 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" event={"ID":"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9","Type":"ContainerStarted","Data":"73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e"} Jul 08 18:29:29.069288 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:29.068978 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:29.070421 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:29.070387 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-87v27" event={"ID":"8ec391ec-594e-43ad-b4a3-44ef948c2f8a","Type":"ContainerStarted","Data":"afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f"} Jul 08 18:29:29.090324 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:29.090280 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" podStartSLOduration=1.277606649 podStartE2EDuration="6.090267325s" totalImagesPullingTime="4.812660676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:29:29.088802004 +0000 UTC m=+703.712389085" watchObservedRunningTime="2026-07-08 18:29:29.090267325 +0000 UTC m=+703.713854404" Jul 08 18:29:29.108215 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:29.108174 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-87v27" podStartSLOduration=1.340247253 podStartE2EDuration="5.108161568s" totalImagesPullingTime="3.767914315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:29:29.106459441 +0000 UTC m=+703.730046523" watchObservedRunningTime="2026-07-08 18:29:29.108161568 +0000 UTC m=+703.731748648" Jul 08 18:29:39.813135 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:39.812993 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nkzzp"] Jul 08 18:29:39.813730 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:39.813240 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" podUID="9d362f8b-a94d-4ce2-acb4-f5a42d6914a9" containerName="limitador" containerID="cri-o://73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e" gracePeriod=30 Jul 08 18:29:39.814579 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:39.814556 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:40.359975 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:40.359953 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:40.442036 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:40.441976 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-config-file\") pod \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\" (UID: \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\") " Jul 08 18:29:40.442136 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:40.442034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc2v8\" (UniqueName: \"kubernetes.io/projected/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-kube-api-access-jc2v8\") pod \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\" (UID: \"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9\") " Jul 08 18:29:40.442304 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:40.442284 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-config-file" (OuterVolumeSpecName: "config-file") pod "9d362f8b-a94d-4ce2-acb4-f5a42d6914a9" (UID: "9d362f8b-a94d-4ce2-acb4-f5a42d6914a9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:29:40.443988 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:40.443964 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-kube-api-access-jc2v8" (OuterVolumeSpecName: "kube-api-access-jc2v8") pod "9d362f8b-a94d-4ce2-acb4-f5a42d6914a9" (UID: "9d362f8b-a94d-4ce2-acb4-f5a42d6914a9"). InnerVolumeSpecName "kube-api-access-jc2v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:29:40.543053 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:40.543020 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc2v8\" (UniqueName: \"kubernetes.io/projected/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-kube-api-access-jc2v8\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:29:40.543053 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:40.543051 2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9-config-file\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:29:41.140941 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.140909 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d362f8b-a94d-4ce2-acb4-f5a42d6914a9" containerID="73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e" exitCode=0 Jul 08 18:29:41.140941 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.140942 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" event={"ID":"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9","Type":"ContainerDied","Data":"73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e"} Jul 08 18:29:41.141494 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.140962 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" Jul 08 18:29:41.141494 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.140977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nkzzp" event={"ID":"9d362f8b-a94d-4ce2-acb4-f5a42d6914a9","Type":"ContainerDied","Data":"09c7e031bd1ba29c2195433709b9d834763298b12aeb97449e12a6347c645a17"} Jul 08 18:29:41.141494 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.140999 2565 scope.go:117] "RemoveContainer" containerID="73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e" Jul 08 18:29:41.150538 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.150522 2565 scope.go:117] "RemoveContainer" containerID="73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e" Jul 08 18:29:41.150793 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:29:41.150773 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e\": container with ID starting with 73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e not found: ID does not exist" containerID="73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e" Jul 08 18:29:41.150859 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.150800 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e"} err="failed to get container status \"73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e\": rpc error: code = NotFound desc = could not find container \"73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e\": container with ID starting with 73d55e9054cd839e2f8e2dbe1e9323615a3fceca4183123a566c6431affcd48e not found: ID does not exist" Jul 08 18:29:41.163449 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.163429 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nkzzp"] Jul 08 18:29:41.167369 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.167345 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nkzzp"] Jul 08 18:29:41.928881 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:41.928852 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d362f8b-a94d-4ce2-acb4-f5a42d6914a9" path="/var/lib/kubelet/pods/9d362f8b-a94d-4ce2-acb4-f5a42d6914a9/volumes" Jul 08 18:29:57.014595 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.014559 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fgrz4"] Jul 08 18:29:57.015256 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.015235 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d362f8b-a94d-4ce2-acb4-f5a42d6914a9" containerName="limitador" Jul 08 18:29:57.015348 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.015258 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d362f8b-a94d-4ce2-acb4-f5a42d6914a9" containerName="limitador" Jul 08 18:29:57.015401 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.015362 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d362f8b-a94d-4ce2-acb4-f5a42d6914a9" containerName="limitador" Jul 08 18:29:57.094942 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.094914 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fgrz4"] Jul 08 18:29:57.095085 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.095008 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fgrz4" Jul 08 18:29:57.171306 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.171277 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkl4r\" (UniqueName: \"kubernetes.io/projected/2ea18433-db09-46d0-8eb9-e3cd0d1e9481-kube-api-access-gkl4r\") pod \"authorino-5d4b67c446-fgrz4\" (UID: \"2ea18433-db09-46d0-8eb9-e3cd0d1e9481\") " pod="kuadrant-system/authorino-5d4b67c446-fgrz4" Jul 08 18:29:57.266472 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.266411 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fgrz4"] Jul 08 18:29:57.266681 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:29:57.266647 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gkl4r], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-fgrz4" podUID="2ea18433-db09-46d0-8eb9-e3cd0d1e9481" Jul 08 18:29:57.272195 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.272166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkl4r\" (UniqueName: \"kubernetes.io/projected/2ea18433-db09-46d0-8eb9-e3cd0d1e9481-kube-api-access-gkl4r\") pod \"authorino-5d4b67c446-fgrz4\" (UID: \"2ea18433-db09-46d0-8eb9-e3cd0d1e9481\") " pod="kuadrant-system/authorino-5d4b67c446-fgrz4" Jul 08 18:29:57.281123 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.281095 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkl4r\" (UniqueName: \"kubernetes.io/projected/2ea18433-db09-46d0-8eb9-e3cd0d1e9481-kube-api-access-gkl4r\") pod \"authorino-5d4b67c446-fgrz4\" (UID: \"2ea18433-db09-46d0-8eb9-e3cd0d1e9481\") " pod="kuadrant-system/authorino-5d4b67c446-fgrz4" Jul 08 18:29:57.384160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.384138 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c6c86679d-qd7ws"] Jul 08 18:29:57.425364 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.425342 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c6c86679d-qd7ws"] Jul 08 18:29:57.425487 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.425421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:29:57.428367 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.428349 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 18:29:57.574382 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.574332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrsl\" (UniqueName: \"kubernetes.io/projected/41f51a43-2001-4911-afe4-b5c05da4af75-kube-api-access-xmrsl\") pod \"authorino-7c6c86679d-qd7ws\" (UID: \"41f51a43-2001-4911-afe4-b5c05da4af75\") " pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:29:57.574473 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.574396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41f51a43-2001-4911-afe4-b5c05da4af75-tls-cert\") pod \"authorino-7c6c86679d-qd7ws\" (UID: \"41f51a43-2001-4911-afe4-b5c05da4af75\") " pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:29:57.675635 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.675611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41f51a43-2001-4911-afe4-b5c05da4af75-tls-cert\") pod \"authorino-7c6c86679d-qd7ws\" (UID: \"41f51a43-2001-4911-afe4-b5c05da4af75\") " pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:29:57.675751 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.675689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrsl\" (UniqueName: \"kubernetes.io/projected/41f51a43-2001-4911-afe4-b5c05da4af75-kube-api-access-xmrsl\") pod \"authorino-7c6c86679d-qd7ws\" (UID: \"41f51a43-2001-4911-afe4-b5c05da4af75\") " pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:29:57.677837 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.677815 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41f51a43-2001-4911-afe4-b5c05da4af75-tls-cert\") pod \"authorino-7c6c86679d-qd7ws\" (UID: \"41f51a43-2001-4911-afe4-b5c05da4af75\") " pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:29:57.684127 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.684101 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrsl\" (UniqueName: \"kubernetes.io/projected/41f51a43-2001-4911-afe4-b5c05da4af75-kube-api-access-xmrsl\") pod \"authorino-7c6c86679d-qd7ws\" (UID: \"41f51a43-2001-4911-afe4-b5c05da4af75\") " pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:29:57.735953 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.735932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:29:57.901685 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:57.901636 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c6c86679d-qd7ws"] Jul 08 18:29:57.903718 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:29:57.903693 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41f51a43_2001_4911_afe4_b5c05da4af75.slice/crio-e03f4260fc7f141390daeff70c1660e59582951643cdb82e887041b4d17e30f1 WatchSource:0}: Error finding container e03f4260fc7f141390daeff70c1660e59582951643cdb82e887041b4d17e30f1: Status 404 returned error can't find the container with id e03f4260fc7f141390daeff70c1660e59582951643cdb82e887041b4d17e30f1 Jul 08 18:29:58.233072 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:58.233002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" event={"ID":"41f51a43-2001-4911-afe4-b5c05da4af75","Type":"ContainerStarted","Data":"e03f4260fc7f141390daeff70c1660e59582951643cdb82e887041b4d17e30f1"} Jul 08 18:29:58.233406 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:58.233092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fgrz4" Jul 08 18:29:58.271340 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:58.271318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fgrz4" Jul 08 18:29:58.382505 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:58.382481 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkl4r\" (UniqueName: \"kubernetes.io/projected/2ea18433-db09-46d0-8eb9-e3cd0d1e9481-kube-api-access-gkl4r\") pod \"2ea18433-db09-46d0-8eb9-e3cd0d1e9481\" (UID: \"2ea18433-db09-46d0-8eb9-e3cd0d1e9481\") " Jul 08 18:29:58.384415 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:58.384395 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea18433-db09-46d0-8eb9-e3cd0d1e9481-kube-api-access-gkl4r" (OuterVolumeSpecName: "kube-api-access-gkl4r") pod "2ea18433-db09-46d0-8eb9-e3cd0d1e9481" (UID: "2ea18433-db09-46d0-8eb9-e3cd0d1e9481"). InnerVolumeSpecName "kube-api-access-gkl4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:29:58.483928 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:58.483880 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkl4r\" (UniqueName: \"kubernetes.io/projected/2ea18433-db09-46d0-8eb9-e3cd0d1e9481-kube-api-access-gkl4r\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:29:59.239804 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.239771 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" event={"ID":"41f51a43-2001-4911-afe4-b5c05da4af75","Type":"ContainerStarted","Data":"ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d"} Jul 08 18:29:59.239804 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.239806 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fgrz4" Jul 08 18:29:59.259044 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.258998 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" podStartSLOduration=1.674842615 podStartE2EDuration="2.258984845s" totalImagesPullingTime="584.14223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:29:59.25734044 +0000 UTC m=+733.880927520" watchObservedRunningTime="2026-07-08 18:29:59.258984845 +0000 UTC m=+733.882571919" Jul 08 18:29:59.283291 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.283259 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fgrz4"] Jul 08 18:29:59.288680 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.288635 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-87v27"] Jul 08 18:29:59.289089 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.288893 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-87v27" podUID="8ec391ec-594e-43ad-b4a3-44ef948c2f8a" containerName="authorino" containerID="cri-o://afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f" gracePeriod=30 Jul 08 18:29:59.292830 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.292198 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fgrz4"] Jul 08 18:29:59.536324 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.536302 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-87v27" Jul 08 18:29:59.692810 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.692783 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/8ec391ec-594e-43ad-b4a3-44ef948c2f8a-kube-api-access-q6ksr\") pod \"8ec391ec-594e-43ad-b4a3-44ef948c2f8a\" (UID: \"8ec391ec-594e-43ad-b4a3-44ef948c2f8a\") " Jul 08 18:29:59.694691 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.694649 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec391ec-594e-43ad-b4a3-44ef948c2f8a-kube-api-access-q6ksr" (OuterVolumeSpecName: "kube-api-access-q6ksr") pod "8ec391ec-594e-43ad-b4a3-44ef948c2f8a" (UID: "8ec391ec-594e-43ad-b4a3-44ef948c2f8a"). InnerVolumeSpecName "kube-api-access-q6ksr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:29:59.793938 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.793886 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/8ec391ec-594e-43ad-b4a3-44ef948c2f8a-kube-api-access-q6ksr\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:29:59.927624 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:29:59.927598 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea18433-db09-46d0-8eb9-e3cd0d1e9481" path="/var/lib/kubelet/pods/2ea18433-db09-46d0-8eb9-e3cd0d1e9481/volumes" Jul 08 18:30:00.246155 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.246127 2565 generic.go:356] "Generic (PLEG): container finished" podID="8ec391ec-594e-43ad-b4a3-44ef948c2f8a" containerID="afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f" exitCode=0 Jul 08 18:30:00.246579 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.246162 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-87v27" Jul 08 18:30:00.246579 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.246223 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-87v27" event={"ID":"8ec391ec-594e-43ad-b4a3-44ef948c2f8a","Type":"ContainerDied","Data":"afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f"} Jul 08 18:30:00.246579 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.246270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-87v27" event={"ID":"8ec391ec-594e-43ad-b4a3-44ef948c2f8a","Type":"ContainerDied","Data":"2e997cccee8c7a460deaa25fb8f0509a781dbce7e785e3060437eb3c28f9e14f"} Jul 08 18:30:00.246579 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.246296 2565 scope.go:117] "RemoveContainer" containerID="afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f" Jul 08 18:30:00.255174 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.255159 2565 scope.go:117] "RemoveContainer" containerID="afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f" Jul 08 18:30:00.255418 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:30:00.255401 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f\": container with ID starting with afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f not found: ID does not exist" containerID="afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f" Jul 08 18:30:00.255459 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.255426 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f"} err="failed to get container status \"afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f\": rpc error: code = NotFound desc = could not find container \"afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f\": container with ID starting with afce4450b21f240ce8f3115129a3f9aeadf0a6356bb2aaeea778d4e55f260c1f not found: ID does not exist" Jul 08 18:30:00.262806 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.262786 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-87v27"] Jul 08 18:30:00.267081 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:00.267062 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-87v27"] Jul 08 18:30:01.928413 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:01.928383 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec391ec-594e-43ad-b4a3-44ef948c2f8a" path="/var/lib/kubelet/pods/8ec391ec-594e-43ad-b4a3-44ef948c2f8a/volumes" Jul 08 18:30:03.838861 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.838818 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d677d787f-lrmlh"] Jul 08 18:30:03.839326 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.839259 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec391ec-594e-43ad-b4a3-44ef948c2f8a" containerName="authorino" Jul 08 18:30:03.839326 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.839271 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec391ec-594e-43ad-b4a3-44ef948c2f8a" containerName="authorino" Jul 08 18:30:03.839440 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.839334 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec391ec-594e-43ad-b4a3-44ef948c2f8a" containerName="authorino" Jul 08 18:30:03.842595 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.842573 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:03.845689 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.845652 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 18:30:03.845802 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.845721 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7hx6d\"" Jul 08 18:30:03.846887 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.846871 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 18:30:03.859792 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.859770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d677d787f-lrmlh"] Jul 08 18:30:03.923914 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.923894 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert\") pod \"maas-controller-6d677d787f-lrmlh\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:03.924027 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:03.923949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8t4\" (UniqueName: \"kubernetes.io/projected/563fd819-7f26-4f64-9d8e-f772e0d469a5-kube-api-access-5d8t4\") pod \"maas-controller-6d677d787f-lrmlh\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:04.012974 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.012949 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787785b8d-5xgxm"] Jul 08 18:30:04.015568 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.015548 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:04.025432 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.025404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert\") pod \"maas-controller-6d677d787f-lrmlh\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:04.025535 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.025491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8t4\" (UniqueName: \"kubernetes.io/projected/563fd819-7f26-4f64-9d8e-f772e0d469a5-kube-api-access-5d8t4\") pod \"maas-controller-6d677d787f-lrmlh\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:04.025601 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:30:04.025529 2565 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 08 18:30:04.025601 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:30:04.025598 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert podName:563fd819-7f26-4f64-9d8e-f772e0d469a5 nodeName:}" failed. No retries permitted until 2026-07-08 18:30:04.525578336 +0000 UTC m=+739.149165409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert") pod "maas-controller-6d677d787f-lrmlh" (UID: "563fd819-7f26-4f64-9d8e-f772e0d469a5") : secret "maas-controller-webhook-cert" not found Jul 08 18:30:04.027038 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.027015 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787785b8d-5xgxm"] Jul 08 18:30:04.038299 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.038268 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8t4\" (UniqueName: \"kubernetes.io/projected/563fd819-7f26-4f64-9d8e-f772e0d469a5-kube-api-access-5d8t4\") pod \"maas-controller-6d677d787f-lrmlh\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:04.126274 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.126216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59dtz\" (UniqueName: \"kubernetes.io/projected/fb270868-0ffc-4992-823f-39573d2e4e8b-kube-api-access-59dtz\") pod \"maas-controller-787785b8d-5xgxm\" (UID: \"fb270868-0ffc-4992-823f-39573d2e4e8b\") " pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:04.126274 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.126248 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb270868-0ffc-4992-823f-39573d2e4e8b-webhook-cert\") pod \"maas-controller-787785b8d-5xgxm\" (UID: \"fb270868-0ffc-4992-823f-39573d2e4e8b\") " pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:04.227025 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.227000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59dtz\" (UniqueName: \"kubernetes.io/projected/fb270868-0ffc-4992-823f-39573d2e4e8b-kube-api-access-59dtz\") pod \"maas-controller-787785b8d-5xgxm\" (UID: \"fb270868-0ffc-4992-823f-39573d2e4e8b\") " pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:04.227116 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.227043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb270868-0ffc-4992-823f-39573d2e4e8b-webhook-cert\") pod \"maas-controller-787785b8d-5xgxm\" (UID: \"fb270868-0ffc-4992-823f-39573d2e4e8b\") " pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:04.229176 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.229160 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb270868-0ffc-4992-823f-39573d2e4e8b-webhook-cert\") pod \"maas-controller-787785b8d-5xgxm\" (UID: \"fb270868-0ffc-4992-823f-39573d2e4e8b\") " pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:04.234793 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.234776 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59dtz\" (UniqueName: \"kubernetes.io/projected/fb270868-0ffc-4992-823f-39573d2e4e8b-kube-api-access-59dtz\") pod \"maas-controller-787785b8d-5xgxm\" (UID: \"fb270868-0ffc-4992-823f-39573d2e4e8b\") " pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:04.326128 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.326112 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:04.460573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.460544 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787785b8d-5xgxm"] Jul 08 18:30:04.460823 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:30:04.460798 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb270868_0ffc_4992_823f_39573d2e4e8b.slice/crio-c8a94046e05df5d90bea1877956ef51db50efe2774795088e11141b143f4eaa0 WatchSource:0}: Error finding container c8a94046e05df5d90bea1877956ef51db50efe2774795088e11141b143f4eaa0: Status 404 returned error can't find the container with id c8a94046e05df5d90bea1877956ef51db50efe2774795088e11141b143f4eaa0 Jul 08 18:30:04.529473 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.529449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert\") pod \"maas-controller-6d677d787f-lrmlh\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:04.531636 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.531617 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert\") pod \"maas-controller-6d677d787f-lrmlh\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:04.753384 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.753360 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:04.883740 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:04.883716 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d677d787f-lrmlh"] Jul 08 18:30:04.884110 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:30:04.884089 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563fd819_7f26_4f64_9d8e_f772e0d469a5.slice/crio-eb60f666657c6a8835f6ca570c03ff2350080ff7894eace21c405940f6b07778 WatchSource:0}: Error finding container eb60f666657c6a8835f6ca570c03ff2350080ff7894eace21c405940f6b07778: Status 404 returned error can't find the container with id eb60f666657c6a8835f6ca570c03ff2350080ff7894eace21c405940f6b07778 Jul 08 18:30:05.278374 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:05.278330 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d677d787f-lrmlh" event={"ID":"563fd819-7f26-4f64-9d8e-f772e0d469a5","Type":"ContainerStarted","Data":"eb60f666657c6a8835f6ca570c03ff2350080ff7894eace21c405940f6b07778"} Jul 08 18:30:05.280419 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:05.280386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787785b8d-5xgxm" event={"ID":"fb270868-0ffc-4992-823f-39573d2e4e8b","Type":"ContainerStarted","Data":"c8a94046e05df5d90bea1877956ef51db50efe2774795088e11141b143f4eaa0"} Jul 08 18:30:07.293641 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:07.293602 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787785b8d-5xgxm" event={"ID":"fb270868-0ffc-4992-823f-39573d2e4e8b","Type":"ContainerStarted","Data":"48f41615d839884d66547474b34e170003fb84bff0a6abaa23afb1e50212be73"} Jul 08 18:30:07.294112 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:07.293696 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:07.295191 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:07.295167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d677d787f-lrmlh" event={"ID":"563fd819-7f26-4f64-9d8e-f772e0d469a5","Type":"ContainerStarted","Data":"dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa"} Jul 08 18:30:07.295322 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:07.295219 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:07.311425 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:07.311385 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787785b8d-5xgxm" podStartSLOduration=2.073444782 podStartE2EDuration="4.311372108s" totalImagesPullingTime="2.237927326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:30:07.31049384 +0000 UTC m=+741.934080920" watchObservedRunningTime="2026-07-08 18:30:07.311372108 +0000 UTC m=+741.934959198" Jul 08 18:30:07.327142 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:07.327105 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d677d787f-lrmlh" podStartSLOduration=2.501402346 podStartE2EDuration="4.327092542s" totalImagesPullingTime="1.825690196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:30:07.326210367 +0000 UTC m=+741.949797446" watchObservedRunningTime="2026-07-08 18:30:07.327092542 +0000 UTC m=+741.950679621" Jul 08 18:30:18.306329 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.306300 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:30:18.306740 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.306371 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:18.370477 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.370442 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d677d787f-lrmlh"] Jul 08 18:30:18.370682 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.370642 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6d677d787f-lrmlh" podUID="563fd819-7f26-4f64-9d8e-f772e0d469a5" containerName="manager" containerID="cri-o://dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa" gracePeriod=30 Jul 08 18:30:18.619167 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.619138 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:18.741897 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.741873 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert\") pod \"563fd819-7f26-4f64-9d8e-f772e0d469a5\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " Jul 08 18:30:18.742010 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.741988 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d8t4\" (UniqueName: \"kubernetes.io/projected/563fd819-7f26-4f64-9d8e-f772e0d469a5-kube-api-access-5d8t4\") pod \"563fd819-7f26-4f64-9d8e-f772e0d469a5\" (UID: \"563fd819-7f26-4f64-9d8e-f772e0d469a5\") " Jul 08 18:30:18.744233 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.744147 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "563fd819-7f26-4f64-9d8e-f772e0d469a5" (UID: "563fd819-7f26-4f64-9d8e-f772e0d469a5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:30:18.744384 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.744324 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563fd819-7f26-4f64-9d8e-f772e0d469a5-kube-api-access-5d8t4" (OuterVolumeSpecName: "kube-api-access-5d8t4") pod "563fd819-7f26-4f64-9d8e-f772e0d469a5" (UID: "563fd819-7f26-4f64-9d8e-f772e0d469a5"). InnerVolumeSpecName "kube-api-access-5d8t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:30:18.843704 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.843650 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/563fd819-7f26-4f64-9d8e-f772e0d469a5-webhook-cert\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:30:18.843704 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:18.843701 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d8t4\" (UniqueName: \"kubernetes.io/projected/563fd819-7f26-4f64-9d8e-f772e0d469a5-kube-api-access-5d8t4\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:30:19.364876 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.364845 2565 generic.go:356] "Generic (PLEG): container finished" podID="563fd819-7f26-4f64-9d8e-f772e0d469a5" containerID="dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa" exitCode=0 Jul 08 18:30:19.364876 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.364878 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d677d787f-lrmlh" event={"ID":"563fd819-7f26-4f64-9d8e-f772e0d469a5","Type":"ContainerDied","Data":"dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa"} Jul 08 18:30:19.365343 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.364902 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d677d787f-lrmlh" Jul 08 18:30:19.365343 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.364913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d677d787f-lrmlh" event={"ID":"563fd819-7f26-4f64-9d8e-f772e0d469a5","Type":"ContainerDied","Data":"eb60f666657c6a8835f6ca570c03ff2350080ff7894eace21c405940f6b07778"} Jul 08 18:30:19.365343 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.364934 2565 scope.go:117] "RemoveContainer" containerID="dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa" Jul 08 18:30:19.375231 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.375025 2565 scope.go:117] "RemoveContainer" containerID="dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa" Jul 08 18:30:19.375301 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:30:19.375282 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa\": container with ID starting with dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa not found: ID does not exist" containerID="dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa" Jul 08 18:30:19.375353 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.375309 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa"} err="failed to get container status \"dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa\": rpc error: code = NotFound desc = could not find container \"dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa\": container with ID starting with dd4340de47adaa87e717ca313a90dd23aee5517ba4403f8b8bf8657a8e6621aa not found: ID does not exist" Jul 08 18:30:19.386751 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.386730 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d677d787f-lrmlh"] Jul 08 18:30:19.390862 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.390842 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d677d787f-lrmlh"] Jul 08 18:30:19.928977 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:19.928956 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563fd819-7f26-4f64-9d8e-f772e0d469a5" path="/var/lib/kubelet/pods/563fd819-7f26-4f64-9d8e-f772e0d469a5/volumes" Jul 08 18:30:37.549240 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.549208 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn"] Jul 08 18:30:37.549853 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.549837 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563fd819-7f26-4f64-9d8e-f772e0d469a5" containerName="manager" Jul 08 18:30:37.549913 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.549856 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="563fd819-7f26-4f64-9d8e-f772e0d469a5" containerName="manager" Jul 08 18:30:37.549998 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.549987 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="563fd819-7f26-4f64-9d8e-f772e0d469a5" containerName="manager" Jul 08 18:30:37.556043 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.556023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:37.559577 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.559545 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 08 18:30:37.559709 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.559575 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 18:30:37.559709 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.559608 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 18:30:37.559921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.559897 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rlgwv\"" Jul 08 18:30:37.561767 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.561745 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn"] Jul 08 18:30:37.592366 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.592348 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9zj\" (UniqueName: \"kubernetes.io/projected/99a9e6c8-68df-4126-a0bb-5d248e3faf79-kube-api-access-5d9zj\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:37.592478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.592406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a9e6c8-68df-4126-a0bb-5d248e3faf79-maas-api-tls\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:37.592532 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.592479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a9e6c8-68df-4126-a0bb-5d248e3faf79-openshift-service-ca\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:37.693040 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.693011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a9e6c8-68df-4126-a0bb-5d248e3faf79-maas-api-tls\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:37.693163 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.693050 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a9e6c8-68df-4126-a0bb-5d248e3faf79-openshift-service-ca\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:37.693211 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:30:37.693158 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 08 18:30:37.693265 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.693222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9zj\" (UniqueName: \"kubernetes.io/projected/99a9e6c8-68df-4126-a0bb-5d248e3faf79-kube-api-access-5d9zj\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:37.693320 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:30:37.693303 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9e6c8-68df-4126-a0bb-5d248e3faf79-maas-api-tls podName:99a9e6c8-68df-4126-a0bb-5d248e3faf79 nodeName:}" failed. No retries permitted until 2026-07-08 18:30:38.193281466 +0000 UTC m=+772.816868523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99a9e6c8-68df-4126-a0bb-5d248e3faf79-maas-api-tls") pod "maas-api-5c9c79bdc-vppwn" (UID: "99a9e6c8-68df-4126-a0bb-5d248e3faf79") : secret "maas-api-serving-cert" not found Jul 08 18:30:37.693714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.693694 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a9e6c8-68df-4126-a0bb-5d248e3faf79-openshift-service-ca\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:37.701635 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:37.701615 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9zj\" (UniqueName: \"kubernetes.io/projected/99a9e6c8-68df-4126-a0bb-5d248e3faf79-kube-api-access-5d9zj\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:38.012218 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.012185 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-jrmnv"] Jul 08 18:30:38.016649 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.016630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:38.019112 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.019095 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 08 18:30:38.019260 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.019241 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6q97r\"" Jul 08 18:30:38.026703 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.026680 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-jrmnv"] Jul 08 18:30:38.040352 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.040319 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jkjzd"] Jul 08 18:30:38.044769 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.044752 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:38.057299 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.057279 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jkjzd"] Jul 08 18:30:38.096062 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.096039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44c6\" (UniqueName: \"kubernetes.io/projected/f7407027-91bb-4bea-892b-ff9f42621617-kube-api-access-m44c6\") pod \"payload-pre-processing-7964457696-jrmnv\" (UID: \"f7407027-91bb-4bea-892b-ff9f42621617\") " pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:38.096184 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.096073 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0001ac5d-2a97-428f-89e4-364bab76ffb2-plugins-config-volume\") pod \"payload-processing-58f86c7798-jkjzd\" (UID: \"0001ac5d-2a97-428f-89e4-364bab76ffb2\") " pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:38.096184 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.096090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26db\" (UniqueName: \"kubernetes.io/projected/0001ac5d-2a97-428f-89e4-364bab76ffb2-kube-api-access-r26db\") pod \"payload-processing-58f86c7798-jkjzd\" (UID: \"0001ac5d-2a97-428f-89e4-364bab76ffb2\") " pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:38.096260 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.096187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7407027-91bb-4bea-892b-ff9f42621617-plugins-config-volume\") pod \"payload-pre-processing-7964457696-jrmnv\" (UID: \"f7407027-91bb-4bea-892b-ff9f42621617\") " pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:38.197436 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.197407 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a9e6c8-68df-4126-a0bb-5d248e3faf79-maas-api-tls\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:38.197580 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.197447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7407027-91bb-4bea-892b-ff9f42621617-plugins-config-volume\") pod \"payload-pre-processing-7964457696-jrmnv\" (UID: \"f7407027-91bb-4bea-892b-ff9f42621617\") " pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:38.197580 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.197536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44c6\" (UniqueName: \"kubernetes.io/projected/f7407027-91bb-4bea-892b-ff9f42621617-kube-api-access-m44c6\") pod \"payload-pre-processing-7964457696-jrmnv\" (UID: \"f7407027-91bb-4bea-892b-ff9f42621617\") " pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:38.197580 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.197569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0001ac5d-2a97-428f-89e4-364bab76ffb2-plugins-config-volume\") pod \"payload-processing-58f86c7798-jkjzd\" (UID: \"0001ac5d-2a97-428f-89e4-364bab76ffb2\") " pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:38.197783 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.197594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26db\" (UniqueName: \"kubernetes.io/projected/0001ac5d-2a97-428f-89e4-364bab76ffb2-kube-api-access-r26db\") pod \"payload-processing-58f86c7798-jkjzd\" (UID: \"0001ac5d-2a97-428f-89e4-364bab76ffb2\") " pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:38.198314 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.198295 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7407027-91bb-4bea-892b-ff9f42621617-plugins-config-volume\") pod \"payload-pre-processing-7964457696-jrmnv\" (UID: \"f7407027-91bb-4bea-892b-ff9f42621617\") " pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:38.198364 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.198303 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0001ac5d-2a97-428f-89e4-364bab76ffb2-plugins-config-volume\") pod \"payload-processing-58f86c7798-jkjzd\" (UID: \"0001ac5d-2a97-428f-89e4-364bab76ffb2\") " pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:38.199987 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.199965 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a9e6c8-68df-4126-a0bb-5d248e3faf79-maas-api-tls\") pod \"maas-api-5c9c79bdc-vppwn\" (UID: \"99a9e6c8-68df-4126-a0bb-5d248e3faf79\") " pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:38.205374 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.205355 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26db\" (UniqueName: \"kubernetes.io/projected/0001ac5d-2a97-428f-89e4-364bab76ffb2-kube-api-access-r26db\") pod \"payload-processing-58f86c7798-jkjzd\" (UID: \"0001ac5d-2a97-428f-89e4-364bab76ffb2\") " pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:38.205457 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.205397 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44c6\" (UniqueName: \"kubernetes.io/projected/f7407027-91bb-4bea-892b-ff9f42621617-kube-api-access-m44c6\") pod \"payload-pre-processing-7964457696-jrmnv\" (UID: \"f7407027-91bb-4bea-892b-ff9f42621617\") " pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:38.329317 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.329243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:38.355155 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.355131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:38.469086 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.469030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:38.489770 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.489595 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-jrmnv"] Jul 08 18:30:38.533084 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.533061 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jkjzd"] Jul 08 18:30:38.534814 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:30:38.534768 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0001ac5d_2a97_428f_89e4_364bab76ffb2.slice/crio-bcbe68abc4067e983faaaf8e0210339646c70d527b73282e7e346725dc9ef9f7 WatchSource:0}: Error finding container bcbe68abc4067e983faaaf8e0210339646c70d527b73282e7e346725dc9ef9f7: Status 404 returned error can't find the container with id bcbe68abc4067e983faaaf8e0210339646c70d527b73282e7e346725dc9ef9f7 Jul 08 18:30:38.694306 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:38.694280 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn"] Jul 08 18:30:38.695674 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:30:38.695636 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a9e6c8_68df_4126_a0bb_5d248e3faf79.slice/crio-6897c737287c1affd1a27e3cf0fbaf06bbdf8d38e227283133685f6d50a8ed5c WatchSource:0}: Error finding container 6897c737287c1affd1a27e3cf0fbaf06bbdf8d38e227283133685f6d50a8ed5c: Status 404 returned error can't find the container with id 6897c737287c1affd1a27e3cf0fbaf06bbdf8d38e227283133685f6d50a8ed5c Jul 08 18:30:39.493886 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:39.493794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" event={"ID":"99a9e6c8-68df-4126-a0bb-5d248e3faf79","Type":"ContainerStarted","Data":"6897c737287c1affd1a27e3cf0fbaf06bbdf8d38e227283133685f6d50a8ed5c"} Jul 08 18:30:39.495759 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:39.495688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" event={"ID":"f7407027-91bb-4bea-892b-ff9f42621617","Type":"ContainerStarted","Data":"540e090d81206c888472c84cf2fd4fb615735d826b0add2281d512e5f57488ff"} Jul 08 18:30:39.498976 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:39.498894 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" event={"ID":"0001ac5d-2a97-428f-89e4-364bab76ffb2","Type":"ContainerStarted","Data":"bcbe68abc4067e983faaaf8e0210339646c70d527b73282e7e346725dc9ef9f7"} Jul 08 18:30:40.507250 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:40.507216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" event={"ID":"f7407027-91bb-4bea-892b-ff9f42621617","Type":"ContainerStarted","Data":"355a197c8cfcd299b9e491666f0eb3557aeb066f190c4c5b35eee0ee0f003d72"} Jul 08 18:30:40.507711 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:40.507311 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:40.508854 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:40.508828 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" event={"ID":"0001ac5d-2a97-428f-89e4-364bab76ffb2","Type":"ContainerStarted","Data":"98f6f26b7ee6cf5ef8d519316be6967274ca9d2e46bc48bca58406ae3750d0d2"} Jul 08 18:30:40.509001 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:40.508895 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:40.527167 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:40.527105 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" podStartSLOduration=1.860997601 podStartE2EDuration="3.527086099s" totalImagesPullingTime="1.666088498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:30:40.524002241 +0000 UTC m=+775.147589322" watchObservedRunningTime="2026-07-08 18:30:40.527086099 +0000 UTC m=+775.150673180" Jul 08 18:30:40.542425 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:40.542331 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" podStartSLOduration=0.90947733 podStartE2EDuration="2.542314256s" totalImagesPullingTime="1.632836926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:30:40.54168523 +0000 UTC m=+775.165272307" watchObservedRunningTime="2026-07-08 18:30:40.542314256 +0000 UTC m=+775.165901337" Jul 08 18:30:41.520389 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:41.520350 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" event={"ID":"99a9e6c8-68df-4126-a0bb-5d248e3faf79","Type":"ContainerStarted","Data":"577d2716bece753d3ad48dd927f5327c97e79b95d6abb719f7023c94a30ed191"} Jul 08 18:30:41.520890 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:41.520861 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:41.539230 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:41.539181 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" podStartSLOduration=2.167302269 podStartE2EDuration="4.539163166s" totalImagesPullingTime="2.371860897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:30:41.538117893 +0000 UTC m=+776.161704974" watchObservedRunningTime="2026-07-08 18:30:41.539163166 +0000 UTC m=+776.162750247" Jul 08 18:30:46.554169 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:46.554140 2565 generic.go:356] "Generic (PLEG): container finished" podID="7857d7c4-d98d-4e46-a602-8f7d5535baad" containerID="d34626e64c824f00fe070fb96feeb7fe893fd80546b846cb7171cd7baeae4a33" exitCode=0 Jul 08 18:30:46.554715 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:46.554218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" event={"ID":"7857d7c4-d98d-4e46-a602-8f7d5535baad","Type":"ContainerDied","Data":"d34626e64c824f00fe070fb96feeb7fe893fd80546b846cb7171cd7baeae4a33"} Jul 08 18:30:46.554715 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:46.554608 2565 scope.go:117] "RemoveContainer" containerID="d34626e64c824f00fe070fb96feeb7fe893fd80546b846cb7171cd7baeae4a33" Jul 08 18:30:46.555560 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:46.555536 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:30:46.555621 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:46.555599 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:30:46.555657 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:46.555632 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:30:47.561612 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:47.561575 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" event={"ID":"7857d7c4-d98d-4e46-a602-8f7d5535baad","Type":"ContainerStarted","Data":"11c427b502d6ad0d4f32cead63ddf53d3ec25b3c0d9a4fb4ef1f9b5b81b91239"} Jul 08 18:30:48.540824 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:48.540788 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c9c79bdc-vppwn" Jul 08 18:30:48.560338 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:48.560305 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:30:48.561279 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:48.561257 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-fbrl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 08 18:30:48.561413 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:48.561308 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" podUID="7857d7c4-d98d-4e46-a602-8f7d5535baad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 08 18:30:49.560630 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:49.560599 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-fbrl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 08 18:30:49.561026 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:49.560653 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" podUID="7857d7c4-d98d-4e46-a602-8f7d5535baad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 08 18:30:50.561348 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:50.561313 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-fbrl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 08 18:30:50.561773 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:50.561375 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" podUID="7857d7c4-d98d-4e46-a602-8f7d5535baad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 08 18:30:51.522714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:51.522687 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-jkjzd" Jul 08 18:30:51.522865 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:51.522757 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-jrmnv" Jul 08 18:30:51.565383 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:51.565352 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:30:51.565921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:51.565896 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:30:51.566627 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:51.566591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fbrl4" Jul 08 18:30:53.769242 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.769211 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4"] Jul 08 18:30:53.772476 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.772459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.775139 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.775122 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 18:30:53.776479 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.776460 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rszwk\"" Jul 08 18:30:53.776479 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.776477 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 18:30:53.776683 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.776464 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 08 18:30:53.784173 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.784154 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4"] Jul 08 18:30:53.872109 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.872078 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.872263 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.872168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.872263 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.872199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.872379 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.872260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbtld\" (UniqueName: \"kubernetes.io/projected/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-kube-api-access-vbtld\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.872379 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.872349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.872486 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.872391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973270 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973421 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973421 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973542 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtld\" (UniqueName: \"kubernetes.io/projected/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-kube-api-access-vbtld\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973542 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973542 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973722 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973653 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973779 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973735 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.973866 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.973842 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.975496 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.975470 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.975927 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.975908 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:53.981705 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:53.981686 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtld\" (UniqueName: \"kubernetes.io/projected/c592aefe-60c3-4bf2-9c3c-26c3e71676b4-kube-api-access-vbtld\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4\" (UID: \"c592aefe-60c3-4bf2-9c3c-26c3e71676b4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:54.084208 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:54.084134 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:30:54.232349 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:54.232319 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4"] Jul 08 18:30:54.233946 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:30:54.233913 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc592aefe_60c3_4bf2_9c3c_26c3e71676b4.slice/crio-316b1dc877830ac0357ff07e6d610707c94bc1fcb4706eab8117a103813a67a0 WatchSource:0}: Error finding container 316b1dc877830ac0357ff07e6d610707c94bc1fcb4706eab8117a103813a67a0: Status 404 returned error can't find the container with id 316b1dc877830ac0357ff07e6d610707c94bc1fcb4706eab8117a103813a67a0 Jul 08 18:30:54.608702 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:54.607987 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" event={"ID":"c592aefe-60c3-4bf2-9c3c-26c3e71676b4","Type":"ContainerStarted","Data":"316b1dc877830ac0357ff07e6d610707c94bc1fcb4706eab8117a103813a67a0"} Jul 08 18:30:55.616100 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:55.616070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" event={"ID":"c592aefe-60c3-4bf2-9c3c-26c3e71676b4","Type":"ContainerStarted","Data":"45583b9a5ce7110a558fca9b970bd2be52285b5ed4c6bc8d31952d701228c136"} Jul 08 18:30:55.634756 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:30:55.634710 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" podStartSLOduration=1.442315219 podStartE2EDuration="2.634695664s" totalImagesPullingTime="1.192380445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:30:55.632339727 +0000 UTC m=+790.255926806" watchObservedRunningTime="2026-07-08 18:30:55.634695664 +0000 UTC m=+790.258282744" Jul 08 18:31:04.085158 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:04.085127 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:31:04.085536 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:04.085220 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:31:04.097516 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:04.097496 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:31:04.677282 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:04.677254 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4" Jul 08 18:31:35.364322 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.364291 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n"] Jul 08 18:31:35.369947 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.369924 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.372690 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.372648 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 18:31:35.379931 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.379819 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n"] Jul 08 18:31:35.443758 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.443728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.443921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.443768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmnj\" (UniqueName: \"kubernetes.io/projected/85654501-4f4c-4055-b389-56421de1500e-kube-api-access-pdmnj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.443921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.443791 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.443921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.443852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.443921 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.443890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85654501-4f4c-4055-b389-56421de1500e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.444077 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.443923 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.544987 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.544952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.545172 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.545010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmnj\" (UniqueName: \"kubernetes.io/projected/85654501-4f4c-4055-b389-56421de1500e-kube-api-access-pdmnj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.545172 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.545045 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.545172 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.545081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.545172 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.545128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85654501-4f4c-4055-b389-56421de1500e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.545398 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.545181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.545456 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.545420 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.545511 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.545460 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.545511 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.545490 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.547294 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.547271 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85654501-4f4c-4055-b389-56421de1500e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.547552 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.547520 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85654501-4f4c-4055-b389-56421de1500e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.553461 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.553435 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmnj\" (UniqueName: \"kubernetes.io/projected/85654501-4f4c-4055-b389-56421de1500e-kube-api-access-pdmnj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n\" (UID: \"85654501-4f4c-4055-b389-56421de1500e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.683577 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.683472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:35.838066 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.838038 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n"] Jul 08 18:31:35.839644 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:31:35.839611 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85654501_4f4c_4055_b389_56421de1500e.slice/crio-723cddfe9916183289f5c95f4eeb9e6bb37548910aa430c685c922e03c0a099b WatchSource:0}: Error finding container 723cddfe9916183289f5c95f4eeb9e6bb37548910aa430c685c922e03c0a099b: Status 404 returned error can't find the container with id 723cddfe9916183289f5c95f4eeb9e6bb37548910aa430c685c922e03c0a099b Jul 08 18:31:35.854201 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:35.854174 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" event={"ID":"85654501-4f4c-4055-b389-56421de1500e","Type":"ContainerStarted","Data":"723cddfe9916183289f5c95f4eeb9e6bb37548910aa430c685c922e03c0a099b"} Jul 08 18:31:36.863415 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:36.863373 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" event={"ID":"85654501-4f4c-4055-b389-56421de1500e","Type":"ContainerStarted","Data":"b584543e2d64c901ad6e6d20a6eae6d768d55168375963597c040ddde09a1c3d"} Jul 08 18:31:36.885073 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:36.885010 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" podStartSLOduration=1.666525773 podStartE2EDuration="1.884991322s" totalImagesPullingTime="218.465549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:31:36.88352845 +0000 UTC m=+831.507115568" watchObservedRunningTime="2026-07-08 18:31:36.884991322 +0000 UTC m=+831.508578401" Jul 08 18:31:45.683640 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:45.683604 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:45.684233 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:45.683691 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:45.696023 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:45.695999 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:31:45.929161 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:31:45.929135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n" Jul 08 18:32:45.793864 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:32:45.793787 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:32:45.796338 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:32:45.796306 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:32:45.796538 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:32:45.796509 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:32:45.798841 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:32:45.798817 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:33:10.687364 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:10.687324 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6c86679d-qd7ws"] Jul 08 18:33:10.687976 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:10.687544 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" podUID="41f51a43-2001-4911-afe4-b5c05da4af75" containerName="authorino" containerID="cri-o://ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d" gracePeriod=30 Jul 08 18:33:10.961835 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:10.961803 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:33:11.121500 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.121469 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41f51a43-2001-4911-afe4-b5c05da4af75-tls-cert\") pod \"41f51a43-2001-4911-afe4-b5c05da4af75\" (UID: \"41f51a43-2001-4911-afe4-b5c05da4af75\") " Jul 08 18:33:11.121684 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.121580 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmrsl\" (UniqueName: \"kubernetes.io/projected/41f51a43-2001-4911-afe4-b5c05da4af75-kube-api-access-xmrsl\") pod \"41f51a43-2001-4911-afe4-b5c05da4af75\" (UID: \"41f51a43-2001-4911-afe4-b5c05da4af75\") " Jul 08 18:33:11.123534 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.123506 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f51a43-2001-4911-afe4-b5c05da4af75-kube-api-access-xmrsl" (OuterVolumeSpecName: "kube-api-access-xmrsl") pod "41f51a43-2001-4911-afe4-b5c05da4af75" (UID: "41f51a43-2001-4911-afe4-b5c05da4af75"). InnerVolumeSpecName "kube-api-access-xmrsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:33:11.132716 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.132691 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f51a43-2001-4911-afe4-b5c05da4af75-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "41f51a43-2001-4911-afe4-b5c05da4af75" (UID: "41f51a43-2001-4911-afe4-b5c05da4af75"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:33:11.223133 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.223066 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41f51a43-2001-4911-afe4-b5c05da4af75-tls-cert\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:33:11.223133 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.223092 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmrsl\" (UniqueName: \"kubernetes.io/projected/41f51a43-2001-4911-afe4-b5c05da4af75-kube-api-access-xmrsl\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:33:11.427401 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.427373 2565 generic.go:356] "Generic (PLEG): container finished" podID="41f51a43-2001-4911-afe4-b5c05da4af75" containerID="ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d" exitCode=0 Jul 08 18:33:11.427547 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.427413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" event={"ID":"41f51a43-2001-4911-afe4-b5c05da4af75","Type":"ContainerDied","Data":"ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d"} Jul 08 18:33:11.427547 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.427427 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" Jul 08 18:33:11.427547 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.427462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6c86679d-qd7ws" event={"ID":"41f51a43-2001-4911-afe4-b5c05da4af75","Type":"ContainerDied","Data":"e03f4260fc7f141390daeff70c1660e59582951643cdb82e887041b4d17e30f1"} Jul 08 18:33:11.427547 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.427488 2565 scope.go:117] "RemoveContainer" containerID="ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d" Jul 08 18:33:11.437787 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.437770 2565 scope.go:117] "RemoveContainer" containerID="ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d" Jul 08 18:33:11.438026 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:33:11.438009 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d\": container with ID starting with ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d not found: ID does not exist" containerID="ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d" Jul 08 18:33:11.438069 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.438036 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d"} err="failed to get container status \"ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d\": rpc error: code = NotFound desc = could not find container \"ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d\": container with ID starting with ac1ae6ac87761deee77a767e9c24f2a87758db6263cf4aa67139873d27d6087d not found: ID does not exist" Jul 08 18:33:11.449552 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.449531 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6c86679d-qd7ws"] Jul 08 18:33:11.453266 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.453243 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c6c86679d-qd7ws"] Jul 08 18:33:11.928182 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:33:11.928155 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f51a43-2001-4911-afe4-b5c05da4af75" path="/var/lib/kubelet/pods/41f51a43-2001-4911-afe4-b5c05da4af75/volumes" Jul 08 18:34:22.632509 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.632432 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787785b8d-5xgxm"] Jul 08 18:34:22.633039 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.632648 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-787785b8d-5xgxm" podUID="fb270868-0ffc-4992-823f-39573d2e4e8b" containerName="manager" containerID="cri-o://48f41615d839884d66547474b34e170003fb84bff0a6abaa23afb1e50212be73" gracePeriod=30 Jul 08 18:34:22.856938 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.856910 2565 generic.go:356] "Generic (PLEG): container finished" podID="fb270868-0ffc-4992-823f-39573d2e4e8b" containerID="48f41615d839884d66547474b34e170003fb84bff0a6abaa23afb1e50212be73" exitCode=0 Jul 08 18:34:22.857097 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.856983 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787785b8d-5xgxm" event={"ID":"fb270868-0ffc-4992-823f-39573d2e4e8b","Type":"ContainerDied","Data":"48f41615d839884d66547474b34e170003fb84bff0a6abaa23afb1e50212be73"} Jul 08 18:34:22.885714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.885651 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:34:22.996556 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.996523 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb270868-0ffc-4992-823f-39573d2e4e8b-webhook-cert\") pod \"fb270868-0ffc-4992-823f-39573d2e4e8b\" (UID: \"fb270868-0ffc-4992-823f-39573d2e4e8b\") " Jul 08 18:34:22.996760 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.996575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59dtz\" (UniqueName: \"kubernetes.io/projected/fb270868-0ffc-4992-823f-39573d2e4e8b-kube-api-access-59dtz\") pod \"fb270868-0ffc-4992-823f-39573d2e4e8b\" (UID: \"fb270868-0ffc-4992-823f-39573d2e4e8b\") " Jul 08 18:34:22.998515 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.998492 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb270868-0ffc-4992-823f-39573d2e4e8b-kube-api-access-59dtz" (OuterVolumeSpecName: "kube-api-access-59dtz") pod "fb270868-0ffc-4992-823f-39573d2e4e8b" (UID: "fb270868-0ffc-4992-823f-39573d2e4e8b"). InnerVolumeSpecName "kube-api-access-59dtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:34:22.998637 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:22.998598 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb270868-0ffc-4992-823f-39573d2e4e8b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fb270868-0ffc-4992-823f-39573d2e4e8b" (UID: "fb270868-0ffc-4992-823f-39573d2e4e8b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:34:23.097933 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:23.097908 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb270868-0ffc-4992-823f-39573d2e4e8b-webhook-cert\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:34:23.097933 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:23.097931 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59dtz\" (UniqueName: \"kubernetes.io/projected/fb270868-0ffc-4992-823f-39573d2e4e8b-kube-api-access-59dtz\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:34:23.864441 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:23.864413 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787785b8d-5xgxm" Jul 08 18:34:23.864940 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:23.864459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787785b8d-5xgxm" event={"ID":"fb270868-0ffc-4992-823f-39573d2e4e8b","Type":"ContainerDied","Data":"c8a94046e05df5d90bea1877956ef51db50efe2774795088e11141b143f4eaa0"} Jul 08 18:34:23.864940 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:23.864522 2565 scope.go:117] "RemoveContainer" containerID="48f41615d839884d66547474b34e170003fb84bff0a6abaa23afb1e50212be73" Jul 08 18:34:23.886070 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:23.886046 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787785b8d-5xgxm"] Jul 08 18:34:23.888577 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:23.888559 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-787785b8d-5xgxm"] Jul 08 18:34:23.932937 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:23.932866 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb270868-0ffc-4992-823f-39573d2e4e8b" path="/var/lib/kubelet/pods/fb270868-0ffc-4992-823f-39573d2e4e8b/volumes" Jul 08 18:34:24.708406 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.708377 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787785b8d-27vss"] Jul 08 18:34:24.708821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.708809 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb270868-0ffc-4992-823f-39573d2e4e8b" containerName="manager" Jul 08 18:34:24.708878 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.708823 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb270868-0ffc-4992-823f-39573d2e4e8b" containerName="manager" Jul 08 18:34:24.708878 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.708850 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f51a43-2001-4911-afe4-b5c05da4af75" containerName="authorino" Jul 08 18:34:24.708878 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.708855 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f51a43-2001-4911-afe4-b5c05da4af75" containerName="authorino" Jul 08 18:34:24.708972 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.708940 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb270868-0ffc-4992-823f-39573d2e4e8b" containerName="manager" Jul 08 18:34:24.708972 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.708951 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f51a43-2001-4911-afe4-b5c05da4af75" containerName="authorino" Jul 08 18:34:24.713364 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.713345 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:24.716040 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.716015 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 18:34:24.717343 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.717134 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7hx6d\"" Jul 08 18:34:24.717343 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.717165 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 18:34:24.721919 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.721889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787785b8d-27vss"] Jul 08 18:34:24.812042 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.812014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lg5\" (UniqueName: \"kubernetes.io/projected/a518d7dc-8080-4d99-8ec9-44b77e901b91-kube-api-access-k9lg5\") pod \"maas-controller-787785b8d-27vss\" (UID: \"a518d7dc-8080-4d99-8ec9-44b77e901b91\") " pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:24.812207 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.812085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a518d7dc-8080-4d99-8ec9-44b77e901b91-webhook-cert\") pod \"maas-controller-787785b8d-27vss\" (UID: \"a518d7dc-8080-4d99-8ec9-44b77e901b91\") " pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:24.913561 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.913531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lg5\" (UniqueName: \"kubernetes.io/projected/a518d7dc-8080-4d99-8ec9-44b77e901b91-kube-api-access-k9lg5\") pod \"maas-controller-787785b8d-27vss\" (UID: \"a518d7dc-8080-4d99-8ec9-44b77e901b91\") " pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:24.913927 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.913597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a518d7dc-8080-4d99-8ec9-44b77e901b91-webhook-cert\") pod \"maas-controller-787785b8d-27vss\" (UID: \"a518d7dc-8080-4d99-8ec9-44b77e901b91\") " pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:24.916196 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.916165 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a518d7dc-8080-4d99-8ec9-44b77e901b91-webhook-cert\") pod \"maas-controller-787785b8d-27vss\" (UID: \"a518d7dc-8080-4d99-8ec9-44b77e901b91\") " pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:24.921938 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:24.921917 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lg5\" (UniqueName: \"kubernetes.io/projected/a518d7dc-8080-4d99-8ec9-44b77e901b91-kube-api-access-k9lg5\") pod \"maas-controller-787785b8d-27vss\" (UID: \"a518d7dc-8080-4d99-8ec9-44b77e901b91\") " pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:25.025920 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:25.025894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:25.167522 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:25.167492 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787785b8d-27vss"] Jul 08 18:34:25.168130 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:34:25.168108 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda518d7dc_8080_4d99_8ec9_44b77e901b91.slice/crio-93060688c879e0c251d1e216bb8c4974467280f12f3f55162ab5339f87ec1bee WatchSource:0}: Error finding container 93060688c879e0c251d1e216bb8c4974467280f12f3f55162ab5339f87ec1bee: Status 404 returned error can't find the container with id 93060688c879e0c251d1e216bb8c4974467280f12f3f55162ab5339f87ec1bee Jul 08 18:34:25.169684 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:25.169641 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:34:25.878406 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:25.878375 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787785b8d-27vss" event={"ID":"a518d7dc-8080-4d99-8ec9-44b77e901b91","Type":"ContainerStarted","Data":"3ab8eee9ad2e1044d34627e1578beec97b355cbda306d39577adda3e271a35d6"} Jul 08 18:34:25.878586 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:25.878424 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787785b8d-27vss" event={"ID":"a518d7dc-8080-4d99-8ec9-44b77e901b91","Type":"ContainerStarted","Data":"93060688c879e0c251d1e216bb8c4974467280f12f3f55162ab5339f87ec1bee"} Jul 08 18:34:25.878586 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:25.878550 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:34:25.896815 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:25.896771 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787785b8d-27vss" podStartSLOduration=1.348355814 podStartE2EDuration="1.89675747s" totalImagesPullingTime="548.401656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:34:25.89501424 +0000 UTC m=+1000.518601335" watchObservedRunningTime="2026-07-08 18:34:25.89675747 +0000 UTC m=+1000.520344550" Jul 08 18:34:36.890549 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:34:36.890523 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787785b8d-27vss" Jul 08 18:37:45.848547 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:37:45.848517 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:37:45.849142 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:37:45.848598 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:37:45.851279 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:37:45.851257 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:37:45.851279 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:37:45.851274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:39:08.222018 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:08.221979 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw"] Jul 08 18:39:08.222592 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:08.222226 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" podUID="26d52641-106b-48ec-9244-e8995c454aec" containerName="manager" containerID="cri-o://1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025" gracePeriod=10 Jul 08 18:39:11.842685 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:11.842634 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" podUID="26d52641-106b-48ec-9244-e8995c454aec" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.52:8081/readyz\": dial tcp 10.132.0.52:8081: connect: connection refused" Jul 08 18:39:11.993142 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:11.993114 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:39:12.176694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.176566 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26d52641-106b-48ec-9244-e8995c454aec-extensions-socket-volume\") pod \"26d52641-106b-48ec-9244-e8995c454aec\" (UID: \"26d52641-106b-48ec-9244-e8995c454aec\") " Jul 08 18:39:12.176694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.176623 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9ncb\" (UniqueName: \"kubernetes.io/projected/26d52641-106b-48ec-9244-e8995c454aec-kube-api-access-r9ncb\") pod \"26d52641-106b-48ec-9244-e8995c454aec\" (UID: \"26d52641-106b-48ec-9244-e8995c454aec\") " Jul 08 18:39:12.177034 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.177009 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d52641-106b-48ec-9244-e8995c454aec-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "26d52641-106b-48ec-9244-e8995c454aec" (UID: "26d52641-106b-48ec-9244-e8995c454aec"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:39:12.178831 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.178805 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d52641-106b-48ec-9244-e8995c454aec-kube-api-access-r9ncb" (OuterVolumeSpecName: "kube-api-access-r9ncb") pod "26d52641-106b-48ec-9244-e8995c454aec" (UID: "26d52641-106b-48ec-9244-e8995c454aec"). InnerVolumeSpecName "kube-api-access-r9ncb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:39:12.277899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.277852 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26d52641-106b-48ec-9244-e8995c454aec-extensions-socket-volume\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:39:12.277899 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.277890 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9ncb\" (UniqueName: \"kubernetes.io/projected/26d52641-106b-48ec-9244-e8995c454aec-kube-api-access-r9ncb\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:39:12.642998 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.642964 2565 generic.go:356] "Generic (PLEG): container finished" podID="26d52641-106b-48ec-9244-e8995c454aec" containerID="1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025" exitCode=0 Jul 08 18:39:12.643192 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.643008 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" Jul 08 18:39:12.643192 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.643033 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" event={"ID":"26d52641-106b-48ec-9244-e8995c454aec","Type":"ContainerDied","Data":"1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025"} Jul 08 18:39:12.643192 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.643076 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw" event={"ID":"26d52641-106b-48ec-9244-e8995c454aec","Type":"ContainerDied","Data":"1e0e4a76f36a4a8128e8e4031027a9c45a75966b031725728e6af22b70cd44ae"} Jul 08 18:39:12.643192 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.643098 2565 scope.go:117] "RemoveContainer" containerID="1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025" Jul 08 18:39:12.654077 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.654057 2565 scope.go:117] "RemoveContainer" containerID="1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025" Jul 08 18:39:12.654392 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:39:12.654371 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025\": container with ID starting with 1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025 not found: ID does not exist" containerID="1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025" Jul 08 18:39:12.654477 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.654405 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025"} err="failed to get container status \"1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025\": rpc error: code = NotFound desc = could not find container \"1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025\": container with ID starting with 1b48f1901a96976673c44be8f6278f1d850177f584c6fc439611e890a94e0025 not found: ID does not exist" Jul 08 18:39:12.665881 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.665844 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw"] Jul 08 18:39:12.669980 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:12.669947 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76mzw"] Jul 08 18:39:13.929119 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:39:13.929080 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d52641-106b-48ec-9244-e8995c454aec" path="/var/lib/kubelet/pods/26d52641-106b-48ec-9244-e8995c454aec/volumes" Jul 08 18:40:09.455458 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.449775 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp"] Jul 08 18:40:09.455458 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.451413 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d52641-106b-48ec-9244-e8995c454aec" containerName="manager" Jul 08 18:40:09.455458 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.451452 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d52641-106b-48ec-9244-e8995c454aec" containerName="manager" Jul 08 18:40:09.455458 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.451776 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d52641-106b-48ec-9244-e8995c454aec" containerName="manager" Jul 08 18:40:09.457619 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.457597 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:09.460968 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.460934 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gc8xq\"" Jul 08 18:40:09.464094 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.464063 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp"] Jul 08 18:40:09.526632 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.526587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce443991-19b1-4404-ac73-de7c475f248b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k8pcp\" (UID: \"ce443991-19b1-4404-ac73-de7c475f248b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:09.526632 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.526642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwcff\" (UniqueName: \"kubernetes.io/projected/ce443991-19b1-4404-ac73-de7c475f248b-kube-api-access-fwcff\") pod \"kuadrant-operator-controller-manager-86878c769f-k8pcp\" (UID: \"ce443991-19b1-4404-ac73-de7c475f248b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:09.628086 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.628042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce443991-19b1-4404-ac73-de7c475f248b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k8pcp\" (UID: \"ce443991-19b1-4404-ac73-de7c475f248b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:09.628086 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.628095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwcff\" (UniqueName: \"kubernetes.io/projected/ce443991-19b1-4404-ac73-de7c475f248b-kube-api-access-fwcff\") pod \"kuadrant-operator-controller-manager-86878c769f-k8pcp\" (UID: \"ce443991-19b1-4404-ac73-de7c475f248b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:09.628460 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.628439 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce443991-19b1-4404-ac73-de7c475f248b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k8pcp\" (UID: \"ce443991-19b1-4404-ac73-de7c475f248b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:09.637751 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.637724 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwcff\" (UniqueName: \"kubernetes.io/projected/ce443991-19b1-4404-ac73-de7c475f248b-kube-api-access-fwcff\") pod \"kuadrant-operator-controller-manager-86878c769f-k8pcp\" (UID: \"ce443991-19b1-4404-ac73-de7c475f248b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:09.771135 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.771090 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:09.930878 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:09.930855 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp"] Jul 08 18:40:09.933326 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:40:09.933296 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce443991_19b1_4404_ac73_de7c475f248b.slice/crio-044541175eef645825c9c1a7e28d984685e173c8b0efd0269b9f1424693df666 WatchSource:0}: Error finding container 044541175eef645825c9c1a7e28d984685e173c8b0efd0269b9f1424693df666: Status 404 returned error can't find the container with id 044541175eef645825c9c1a7e28d984685e173c8b0efd0269b9f1424693df666 Jul 08 18:40:10.017474 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:10.017437 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" event={"ID":"ce443991-19b1-4404-ac73-de7c475f248b","Type":"ContainerStarted","Data":"9105379fd54f0ccc70527362d8013f99fdb0c97840b82cd896639e8ace925234"} Jul 08 18:40:10.017729 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:10.017502 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" event={"ID":"ce443991-19b1-4404-ac73-de7c475f248b","Type":"ContainerStarted","Data":"044541175eef645825c9c1a7e28d984685e173c8b0efd0269b9f1424693df666"} Jul 08 18:40:10.017729 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:10.017692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:40:10.037294 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:10.037172 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" podStartSLOduration=1.037153493 podStartE2EDuration="1.037153493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:10.037089105 +0000 UTC m=+1344.660676258" watchObservedRunningTime="2026-07-08 18:40:10.037153493 +0000 UTC m=+1344.660740577" Jul 08 18:40:21.026539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:40:21.026510 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8pcp" Jul 08 18:42:45.890150 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:42:45.890126 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:42:45.892303 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:42:45.892285 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:42:45.893319 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:42:45.893300 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:42:45.895483 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:42:45.895468 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:45:00.150209 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.150174 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6"] Jul 08 18:45:00.154221 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.154204 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" Jul 08 18:45:00.168357 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.168338 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6"] Jul 08 18:45:00.308713 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.308683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8np\" (UniqueName: \"kubernetes.io/projected/3a6c2c53-9942-4505-8d61-93df91248c6c-kube-api-access-fm8np\") pod \"maas-api-key-cleanup-29725605-t79c6\" (UID: \"3a6c2c53-9942-4505-8d61-93df91248c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" Jul 08 18:45:00.409283 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.409221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8np\" (UniqueName: \"kubernetes.io/projected/3a6c2c53-9942-4505-8d61-93df91248c6c-kube-api-access-fm8np\") pod \"maas-api-key-cleanup-29725605-t79c6\" (UID: \"3a6c2c53-9942-4505-8d61-93df91248c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" Jul 08 18:45:00.418186 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.418152 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8np\" (UniqueName: \"kubernetes.io/projected/3a6c2c53-9942-4505-8d61-93df91248c6c-kube-api-access-fm8np\") pod \"maas-api-key-cleanup-29725605-t79c6\" (UID: \"3a6c2c53-9942-4505-8d61-93df91248c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" Jul 08 18:45:00.464396 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.464373 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" Jul 08 18:45:00.602804 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.602772 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6"] Jul 08 18:45:00.604048 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:45:00.604021 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6c2c53_9942_4505_8d61_93df91248c6c.slice/crio-ed408b123ff64c9a7fdbd3727b37817b7ce206afb2541581cec9607377799be7 WatchSource:0}: Error finding container ed408b123ff64c9a7fdbd3727b37817b7ce206afb2541581cec9607377799be7: Status 404 returned error can't find the container with id ed408b123ff64c9a7fdbd3727b37817b7ce206afb2541581cec9607377799be7 Jul 08 18:45:00.605904 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.605887 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:45:00.761754 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:00.761712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" event={"ID":"3a6c2c53-9942-4505-8d61-93df91248c6c","Type":"ContainerStarted","Data":"ed408b123ff64c9a7fdbd3727b37817b7ce206afb2541581cec9607377799be7"} Jul 08 18:45:04.789117 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:04.789084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" event={"ID":"3a6c2c53-9942-4505-8d61-93df91248c6c","Type":"ContainerStarted","Data":"0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159"} Jul 08 18:45:04.807721 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:45:04.807673 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" podStartSLOduration=1.407767547 podStartE2EDuration="4.807643947s" totalImagesPullingTime="3.3998764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:45:04.805704979 +0000 UTC m=+1639.429292049" watchObservedRunningTime="2026-07-08 18:45:04.807643947 +0000 UTC m=+1639.431231026" Jul 08 18:46:00.011349 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:00.011316 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6"] Jul 08 18:46:00.011860 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:00.011516 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" podUID="3a6c2c53-9942-4505-8d61-93df91248c6c" containerName="cleanup" containerID="cri-o://0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159" gracePeriod=30 Jul 08 18:46:30.205356 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.205328 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" Jul 08 18:46:30.249312 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.249284 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm8np\" (UniqueName: \"kubernetes.io/projected/3a6c2c53-9942-4505-8d61-93df91248c6c-kube-api-access-fm8np\") pod \"3a6c2c53-9942-4505-8d61-93df91248c6c\" (UID: \"3a6c2c53-9942-4505-8d61-93df91248c6c\") " Jul 08 18:46:30.251334 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.251309 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6c2c53-9942-4505-8d61-93df91248c6c-kube-api-access-fm8np" (OuterVolumeSpecName: "kube-api-access-fm8np") pod "3a6c2c53-9942-4505-8d61-93df91248c6c" (UID: "3a6c2c53-9942-4505-8d61-93df91248c6c"). InnerVolumeSpecName "kube-api-access-fm8np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:46:30.325074 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.325022 2565 generic.go:356] "Generic (PLEG): container finished" podID="3a6c2c53-9942-4505-8d61-93df91248c6c" containerID="0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159" exitCode=137 Jul 08 18:46:30.325167 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.325074 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" Jul 08 18:46:30.325167 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.325116 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" event={"ID":"3a6c2c53-9942-4505-8d61-93df91248c6c","Type":"ContainerDied","Data":"0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159"} Jul 08 18:46:30.325240 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.325173 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6" event={"ID":"3a6c2c53-9942-4505-8d61-93df91248c6c","Type":"ContainerDied","Data":"ed408b123ff64c9a7fdbd3727b37817b7ce206afb2541581cec9607377799be7"} Jul 08 18:46:30.325240 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.325201 2565 scope.go:117] "RemoveContainer" containerID="0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159" Jul 08 18:46:30.334599 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.334585 2565 scope.go:117] "RemoveContainer" containerID="0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159" Jul 08 18:46:30.334838 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:46:30.334819 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159\": container with ID starting with 0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159 not found: ID does not exist" containerID="0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159" Jul 08 18:46:30.334907 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.334849 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159"} err="failed to get container status \"0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159\": rpc error: code = NotFound desc = could not find container \"0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159\": container with ID starting with 0fe7379a50cac0ab1ae856120cd1f3ecf14b9ec475f47b5d28ea7471b6c97159 not found: ID does not exist" Jul 08 18:46:30.347321 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.347300 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6"] Jul 08 18:46:30.349919 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.349896 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725605-t79c6"] Jul 08 18:46:30.350153 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:30.350136 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm8np\" (UniqueName: \"kubernetes.io/projected/3a6c2c53-9942-4505-8d61-93df91248c6c-kube-api-access-fm8np\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:46:31.928100 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:31.928071 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6c2c53-9942-4505-8d61-93df91248c6c" path="/var/lib/kubelet/pods/3a6c2c53-9942-4505-8d61-93df91248c6c/volumes" Jul 08 18:46:40.124239 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.124199 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx"] Jul 08 18:46:40.124946 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.124921 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6c2c53-9942-4505-8d61-93df91248c6c" containerName="cleanup" Jul 08 18:46:40.124946 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.124946 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6c2c53-9942-4505-8d61-93df91248c6c" containerName="cleanup" Jul 08 18:46:40.125145 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.125067 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6c2c53-9942-4505-8d61-93df91248c6c" containerName="cleanup" Jul 08 18:46:40.128552 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.128532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.131160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.131136 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-40bf37b7-serving-cert\"" Jul 08 18:46:40.136307 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.136286 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx"] Jul 08 18:46:40.233250 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.233222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5txz\" (UniqueName: \"kubernetes.io/projected/345dc2bf-84f5-4d06-ae56-037500869f51-kube-api-access-v5txz\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.233376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.233267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345dc2bf-84f5-4d06-ae56-037500869f51-openshift-service-ca\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.233376 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.233342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345dc2bf-84f5-4d06-ae56-037500869f51-maas-api-tls\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.334560 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.334532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5txz\" (UniqueName: \"kubernetes.io/projected/345dc2bf-84f5-4d06-ae56-037500869f51-kube-api-access-v5txz\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.334734 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.334572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345dc2bf-84f5-4d06-ae56-037500869f51-openshift-service-ca\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.334734 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.334635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345dc2bf-84f5-4d06-ae56-037500869f51-maas-api-tls\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.335282 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.335258 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345dc2bf-84f5-4d06-ae56-037500869f51-openshift-service-ca\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.337144 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.337122 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345dc2bf-84f5-4d06-ae56-037500869f51-maas-api-tls\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.344691 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.344641 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5txz\" (UniqueName: \"kubernetes.io/projected/345dc2bf-84f5-4d06-ae56-037500869f51-kube-api-access-v5txz\") pod \"maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.442425 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.442356 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:40.582230 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:40.582202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx"] Jul 08 18:46:40.583204 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:46:40.583177 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345dc2bf_84f5_4d06_ae56_037500869f51.slice/crio-c545bf99d94eff7df1e8b8ed6aed0736d453688d7f9431968a7c2443735a3379 WatchSource:0}: Error finding container c545bf99d94eff7df1e8b8ed6aed0736d453688d7f9431968a7c2443735a3379: Status 404 returned error can't find the container with id c545bf99d94eff7df1e8b8ed6aed0736d453688d7f9431968a7c2443735a3379 Jul 08 18:46:41.393942 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:41.393910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" event={"ID":"345dc2bf-84f5-4d06-ae56-037500869f51","Type":"ContainerStarted","Data":"a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0"} Jul 08 18:46:41.394331 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:41.393962 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" event={"ID":"345dc2bf-84f5-4d06-ae56-037500869f51","Type":"ContainerStarted","Data":"c545bf99d94eff7df1e8b8ed6aed0736d453688d7f9431968a7c2443735a3379"} Jul 08 18:46:41.394331 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:41.393983 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:41.411668 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:41.411623 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" podStartSLOduration=0.773162186 podStartE2EDuration="1.411605773s" totalImagesPullingTime="638.443587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:41.410198277 +0000 UTC m=+1736.033785385" watchObservedRunningTime="2026-07-08 18:46:41.411605773 +0000 UTC m=+1736.035192883" Jul 08 18:46:45.826920 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:45.826886 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx"] Jul 08 18:46:45.827750 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:45.827717 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" podUID="345dc2bf-84f5-4d06-ae56-037500869f51" containerName="maas-api" containerID="cri-o://a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0" gracePeriod=30 Jul 08 18:46:46.088383 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.088327 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:46.184079 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.184051 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345dc2bf-84f5-4d06-ae56-037500869f51-maas-api-tls\") pod \"345dc2bf-84f5-4d06-ae56-037500869f51\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " Jul 08 18:46:46.184227 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.184084 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5txz\" (UniqueName: \"kubernetes.io/projected/345dc2bf-84f5-4d06-ae56-037500869f51-kube-api-access-v5txz\") pod \"345dc2bf-84f5-4d06-ae56-037500869f51\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " Jul 08 18:46:46.184227 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.184166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345dc2bf-84f5-4d06-ae56-037500869f51-openshift-service-ca\") pod \"345dc2bf-84f5-4d06-ae56-037500869f51\" (UID: \"345dc2bf-84f5-4d06-ae56-037500869f51\") " Jul 08 18:46:46.184518 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.184478 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345dc2bf-84f5-4d06-ae56-037500869f51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "345dc2bf-84f5-4d06-ae56-037500869f51" (UID: "345dc2bf-84f5-4d06-ae56-037500869f51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:46:46.186161 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.186139 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345dc2bf-84f5-4d06-ae56-037500869f51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "345dc2bf-84f5-4d06-ae56-037500869f51" (UID: "345dc2bf-84f5-4d06-ae56-037500869f51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:46:46.186247 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.186166 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345dc2bf-84f5-4d06-ae56-037500869f51-kube-api-access-v5txz" (OuterVolumeSpecName: "kube-api-access-v5txz") pod "345dc2bf-84f5-4d06-ae56-037500869f51" (UID: "345dc2bf-84f5-4d06-ae56-037500869f51"). InnerVolumeSpecName "kube-api-access-v5txz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:46:46.285705 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.285683 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345dc2bf-84f5-4d06-ae56-037500869f51-openshift-service-ca\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:46:46.285705 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.285704 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345dc2bf-84f5-4d06-ae56-037500869f51-maas-api-tls\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:46:46.285843 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.285714 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5txz\" (UniqueName: \"kubernetes.io/projected/345dc2bf-84f5-4d06-ae56-037500869f51-kube-api-access-v5txz\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:46:46.424785 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.424702 2565 generic.go:356] "Generic (PLEG): container finished" podID="345dc2bf-84f5-4d06-ae56-037500869f51" containerID="a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0" exitCode=0 Jul 08 18:46:46.424785 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.424764 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" Jul 08 18:46:46.424957 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.424789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" event={"ID":"345dc2bf-84f5-4d06-ae56-037500869f51","Type":"ContainerDied","Data":"a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0"} Jul 08 18:46:46.424957 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.424861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx" event={"ID":"345dc2bf-84f5-4d06-ae56-037500869f51","Type":"ContainerDied","Data":"c545bf99d94eff7df1e8b8ed6aed0736d453688d7f9431968a7c2443735a3379"} Jul 08 18:46:46.424957 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.424895 2565 scope.go:117] "RemoveContainer" containerID="a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0" Jul 08 18:46:46.438517 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.438499 2565 scope.go:117] "RemoveContainer" containerID="a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0" Jul 08 18:46:46.438780 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:46:46.438762 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0\": container with ID starting with a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0 not found: ID does not exist" containerID="a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0" Jul 08 18:46:46.438841 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.438789 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0"} err="failed to get container status \"a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0\": rpc error: code = NotFound desc = could not find container \"a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0\": container with ID starting with a095032e0c389cdfd02186358d9dcd89578cd4273c3312c103b703a90e5e90b0 not found: ID does not exist" Jul 08 18:46:46.449350 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.449326 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx"] Jul 08 18:46:46.453199 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:46.453180 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40bf37b7-6dd846c989-jvmgx"] Jul 08 18:46:47.928822 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:47.928788 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="345dc2bf-84f5-4d06-ae56-037500869f51" path="/var/lib/kubelet/pods/345dc2bf-84f5-4d06-ae56-037500869f51/volumes" Jul 08 18:46:51.815871 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.815837 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs"] Jul 08 18:46:51.816503 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.816481 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="345dc2bf-84f5-4d06-ae56-037500869f51" containerName="maas-api" Jul 08 18:46:51.816687 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.816656 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="345dc2bf-84f5-4d06-ae56-037500869f51" containerName="maas-api" Jul 08 18:46:51.816963 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.816943 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="345dc2bf-84f5-4d06-ae56-037500869f51" containerName="maas-api" Jul 08 18:46:51.821459 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.820826 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.824530 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.824506 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-68a2310a-openshift-default-dockercfg-9z8jp\"" Jul 08 18:46:51.845708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.845683 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs"] Jul 08 18:46:51.936590 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-token\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.936590 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-socket\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.936803 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-credential-socket\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.936803 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936713 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-data\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.936803 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-certs\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.936803 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-envoy\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.936959 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-podinfo\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.936959 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbsqf\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-kube-api-access-fbsqf\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:51.937033 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:51.936993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94964b2-e144-47a6-ba8f-ff009fb05444-istiod-ca-cert\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037534 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-credential-socket\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037716 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-data\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037716 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-certs\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037716 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-envoy\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037884 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-podinfo\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037884 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbsqf\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-kube-api-access-fbsqf\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037884 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94964b2-e144-47a6-ba8f-ff009fb05444-istiod-ca-cert\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037884 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-token\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.037884 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-socket\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.038156 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.037985 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-data\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.038156 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.038025 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-credential-socket\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.038296 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.038214 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-certs\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.038375 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.038359 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-socket\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.038547 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.038529 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94964b2-e144-47a6-ba8f-ff009fb05444-istiod-ca-cert\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.040395 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.040372 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-envoy\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.040559 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.040542 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-podinfo\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.045475 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.045450 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-token\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.046468 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.046445 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbsqf\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-kube-api-access-fbsqf\") pod \"e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.132746 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.132692 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:52.277920 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.277893 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs"] Jul 08 18:46:52.279760 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:46:52.279734 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94964b2_e144_47a6_ba8f_ff009fb05444.slice/crio-a72b4c6713d7788390d8e77352715aaf18491a7e5b0d2ea1ac8b353c0a11b31f WatchSource:0}: Error finding container a72b4c6713d7788390d8e77352715aaf18491a7e5b0d2ea1ac8b353c0a11b31f: Status 404 returned error can't find the container with id a72b4c6713d7788390d8e77352715aaf18491a7e5b0d2ea1ac8b353c0a11b31f Jul 08 18:46:52.282162 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.282131 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:46:52.282243 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.282198 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:46:52.282243 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.282228 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 08 18:46:52.460277 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.460197 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" event={"ID":"b94964b2-e144-47a6-ba8f-ff009fb05444","Type":"ContainerStarted","Data":"27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f"} Jul 08 18:46:52.460277 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.460263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" event={"ID":"b94964b2-e144-47a6-ba8f-ff009fb05444","Type":"ContainerStarted","Data":"a72b4c6713d7788390d8e77352715aaf18491a7e5b0d2ea1ac8b353c0a11b31f"} Jul 08 18:46:52.480412 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:52.480354 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" podStartSLOduration=1.480339423 podStartE2EDuration="1.480339423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:52.479386096 +0000 UTC m=+1747.102973175" watchObservedRunningTime="2026-07-08 18:46:52.480339423 +0000 UTC m=+1747.103926566" Jul 08 18:46:53.133726 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:53.133691 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:53.139092 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:53.139070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:53.466113 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:53.466038 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:46:53.467046 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:46:53.467024 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:47:04.233331 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:04.233289 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs"] Jul 08 18:47:04.233895 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:04.233500 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" podUID="b94964b2-e144-47a6-ba8f-ff009fb05444" containerName="istio-proxy" containerID="cri-o://27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f" gracePeriod=30 Jul 08 18:47:07.492581 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.492555 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:47:07.560365 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.560293 2565 generic.go:356] "Generic (PLEG): container finished" podID="b94964b2-e144-47a6-ba8f-ff009fb05444" containerID="27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f" exitCode=0 Jul 08 18:47:07.560365 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.560338 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" Jul 08 18:47:07.560545 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.560380 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" event={"ID":"b94964b2-e144-47a6-ba8f-ff009fb05444","Type":"ContainerDied","Data":"27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f"} Jul 08 18:47:07.560545 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.560435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs" event={"ID":"b94964b2-e144-47a6-ba8f-ff009fb05444","Type":"ContainerDied","Data":"a72b4c6713d7788390d8e77352715aaf18491a7e5b0d2ea1ac8b353c0a11b31f"} Jul 08 18:47:07.560545 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.560462 2565 scope.go:117] "RemoveContainer" containerID="27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f" Jul 08 18:47:07.569507 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.569492 2565 scope.go:117] "RemoveContainer" containerID="27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f" Jul 08 18:47:07.569776 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:47:07.569757 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f\": container with ID starting with 27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f not found: ID does not exist" containerID="27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f" Jul 08 18:47:07.569857 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.569786 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f"} err="failed to get container status \"27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f\": rpc error: code = NotFound desc = could not find container \"27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f\": container with ID starting with 27fe5396e51e0d18294c2aba4e456d8e8b2b85900cf07f15136fc9bfc8fae32f not found: ID does not exist" Jul 08 18:47:07.577144 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577127 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-token\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.577217 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-podinfo\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.577217 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577196 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-data\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.577365 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577351 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-socket\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.577470 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577451 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94964b2-e144-47a6-ba8f-ff009fb05444-istiod-ca-cert\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.577537 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577522 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-envoy\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.577595 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577561 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbsqf\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-kube-api-access-fbsqf\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.577650 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577589 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:47:07.577650 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577599 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-certs\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.577786 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577647 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-credential-socket\") pod \"b94964b2-e144-47a6-ba8f-ff009fb05444\" (UID: \"b94964b2-e144-47a6-ba8f-ff009fb05444\") " Jul 08 18:47:07.578085 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.577952 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:47:07.578085 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.578046 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-certs\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.578085 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.578065 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-workload-socket\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.578085 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.578072 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94964b2-e144-47a6-ba8f-ff009fb05444-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:07.578318 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.578185 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:47:07.578318 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.578234 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-data" (OuterVolumeSpecName: "istio-data") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:47:07.579309 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.579287 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:47:07.579718 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.579693 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-kube-api-access-fbsqf" (OuterVolumeSpecName: "kube-api-access-fbsqf") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "kube-api-access-fbsqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:47:07.579805 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.579780 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-token" (OuterVolumeSpecName: "istio-token") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:47:07.579901 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.579880 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b94964b2-e144-47a6-ba8f-ff009fb05444" (UID: "b94964b2-e144-47a6-ba8f-ff009fb05444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:47:07.678615 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.678589 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-credential-socket\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.678615 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.678616 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-token\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.678821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.678629 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-podinfo\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.678821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.678641 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-data\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.678821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.678654 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94964b2-e144-47a6-ba8f-ff009fb05444-istiod-ca-cert\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.678821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.678683 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94964b2-e144-47a6-ba8f-ff009fb05444-istio-envoy\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.678821 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.678693 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbsqf\" (UniqueName: \"kubernetes.io/projected/b94964b2-e144-47a6-ba8f-ff009fb05444-kube-api-access-fbsqf\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:47:07.884375 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.884351 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs"] Jul 08 18:47:07.887967 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.887948 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-68a2310a-openshift-default-86bf54674c-48qjs"] Jul 08 18:47:07.927964 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:07.927941 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94964b2-e144-47a6-ba8f-ff009fb05444" path="/var/lib/kubelet/pods/b94964b2-e144-47a6-ba8f-ff009fb05444/volumes" Jul 08 18:47:45.937427 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:45.937393 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:47:45.940621 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:45.940597 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:47:45.943292 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:45.943271 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:47:45.945378 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:47:45.945358 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:52:29.651124 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.651086 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp"] Jul 08 18:52:29.651880 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.651861 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94964b2-e144-47a6-ba8f-ff009fb05444" containerName="istio-proxy" Jul 08 18:52:29.651880 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.651881 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94964b2-e144-47a6-ba8f-ff009fb05444" containerName="istio-proxy" Jul 08 18:52:29.652031 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.652008 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94964b2-e144-47a6-ba8f-ff009fb05444" containerName="istio-proxy" Jul 08 18:52:29.655786 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.655764 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:29.658383 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.658362 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-200f0c-serving-cert\"" Jul 08 18:52:29.665158 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.665132 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp"] Jul 08 18:52:29.777097 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.777067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:29.777239 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.777106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f651a3-1ae2-42b7-8967-d7994f7dbd18-openshift-service-ca\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:29.777239 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.777177 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lk9h\" (UniqueName: \"kubernetes.io/projected/46f651a3-1ae2-42b7-8967-d7994f7dbd18-kube-api-access-7lk9h\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:29.878466 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.878438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lk9h\" (UniqueName: \"kubernetes.io/projected/46f651a3-1ae2-42b7-8967-d7994f7dbd18-kube-api-access-7lk9h\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:29.878591 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.878532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:29.878591 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.878574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f651a3-1ae2-42b7-8967-d7994f7dbd18-openshift-service-ca\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:29.878707 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:52:29.878687 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-serving-cert: secret "maas-api-e2e-inf-a-200f0c-serving-cert" not found Jul 08 18:52:29.878764 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:52:29.878753 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls podName:46f651a3-1ae2-42b7-8967-d7994f7dbd18 nodeName:}" failed. No retries permitted until 2026-07-08 18:52:30.378737697 +0000 UTC m=+2085.002324755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls") pod "maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" (UID: "46f651a3-1ae2-42b7-8967-d7994f7dbd18") : secret "maas-api-e2e-inf-a-200f0c-serving-cert" not found Jul 08 18:52:29.879311 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.879292 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f651a3-1ae2-42b7-8967-d7994f7dbd18-openshift-service-ca\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:29.888514 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:29.888492 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lk9h\" (UniqueName: \"kubernetes.io/projected/46f651a3-1ae2-42b7-8967-d7994f7dbd18-kube-api-access-7lk9h\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:30.382267 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:30.382235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:30.384593 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:30.384564 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls\") pod \"maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:30.569587 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:30.569560 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:30.719477 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:30.719446 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp"] Jul 08 18:52:30.720386 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:52:30.720359 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f651a3_1ae2_42b7_8967_d7994f7dbd18.slice/crio-adefa5e0c9f7de97b4b880073f1f00af59b97de77c1a6e2a9d2d299f113bb1e4 WatchSource:0}: Error finding container adefa5e0c9f7de97b4b880073f1f00af59b97de77c1a6e2a9d2d299f113bb1e4: Status 404 returned error can't find the container with id adefa5e0c9f7de97b4b880073f1f00af59b97de77c1a6e2a9d2d299f113bb1e4 Jul 08 18:52:30.721705 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:30.721655 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:52:31.491933 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:31.491901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" event={"ID":"46f651a3-1ae2-42b7-8967-d7994f7dbd18","Type":"ContainerStarted","Data":"1cdb7a2871db6a61bccedb2baf5e22f93481fa68f6d464a0538d89eaf714a830"} Jul 08 18:52:31.492101 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:31.491956 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" event={"ID":"46f651a3-1ae2-42b7-8967-d7994f7dbd18","Type":"ContainerStarted","Data":"adefa5e0c9f7de97b4b880073f1f00af59b97de77c1a6e2a9d2d299f113bb1e4"} Jul 08 18:52:31.492165 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:31.492100 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:31.511758 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:31.511716 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" podStartSLOduration=2.08269877 podStartE2EDuration="2.511698369s" totalImagesPullingTime="428.999599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:52:31.510118246 +0000 UTC m=+2086.133705327" watchObservedRunningTime="2026-07-08 18:52:31.511698369 +0000 UTC m=+2086.135285456" Jul 08 18:52:37.503267 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:37.503235 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:52:45.979320 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:45.979294 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:52:45.981598 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:45.981576 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:52:45.985457 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:45.985441 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:52:45.987442 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:52:45.987425 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:53:57.817607 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:57.817500 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp"] Jul 08 18:53:57.818218 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:57.817768 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" podUID="46f651a3-1ae2-42b7-8967-d7994f7dbd18" containerName="maas-api" containerID="cri-o://1cdb7a2871db6a61bccedb2baf5e22f93481fa68f6d464a0538d89eaf714a830" gracePeriod=30 Jul 08 18:53:58.020771 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.020735 2565 generic.go:356] "Generic (PLEG): container finished" podID="46f651a3-1ae2-42b7-8967-d7994f7dbd18" containerID="1cdb7a2871db6a61bccedb2baf5e22f93481fa68f6d464a0538d89eaf714a830" exitCode=0 Jul 08 18:53:58.020952 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.020792 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" event={"ID":"46f651a3-1ae2-42b7-8967-d7994f7dbd18","Type":"ContainerDied","Data":"1cdb7a2871db6a61bccedb2baf5e22f93481fa68f6d464a0538d89eaf714a830"} Jul 08 18:53:58.079843 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.079786 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:53:58.177866 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.177837 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls\") pod \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " Jul 08 18:53:58.178030 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.177901 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f651a3-1ae2-42b7-8967-d7994f7dbd18-openshift-service-ca\") pod \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " Jul 08 18:53:58.178030 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.177945 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lk9h\" (UniqueName: \"kubernetes.io/projected/46f651a3-1ae2-42b7-8967-d7994f7dbd18-kube-api-access-7lk9h\") pod \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\" (UID: \"46f651a3-1ae2-42b7-8967-d7994f7dbd18\") " Jul 08 18:53:58.178319 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.178292 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f651a3-1ae2-42b7-8967-d7994f7dbd18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46f651a3-1ae2-42b7-8967-d7994f7dbd18" (UID: "46f651a3-1ae2-42b7-8967-d7994f7dbd18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:53:58.179945 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.179918 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46f651a3-1ae2-42b7-8967-d7994f7dbd18" (UID: "46f651a3-1ae2-42b7-8967-d7994f7dbd18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:53:58.180037 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.179956 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f651a3-1ae2-42b7-8967-d7994f7dbd18-kube-api-access-7lk9h" (OuterVolumeSpecName: "kube-api-access-7lk9h") pod "46f651a3-1ae2-42b7-8967-d7994f7dbd18" (UID: "46f651a3-1ae2-42b7-8967-d7994f7dbd18"). InnerVolumeSpecName "kube-api-access-7lk9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:53:58.278730 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.278704 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f651a3-1ae2-42b7-8967-d7994f7dbd18-openshift-service-ca\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:53:58.278730 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.278724 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lk9h\" (UniqueName: \"kubernetes.io/projected/46f651a3-1ae2-42b7-8967-d7994f7dbd18-kube-api-access-7lk9h\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:53:58.278857 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:58.278735 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f651a3-1ae2-42b7-8967-d7994f7dbd18-maas-api-tls\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:53:59.027676 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:59.027645 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" Jul 08 18:53:59.028073 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:59.027685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp" event={"ID":"46f651a3-1ae2-42b7-8967-d7994f7dbd18","Type":"ContainerDied","Data":"adefa5e0c9f7de97b4b880073f1f00af59b97de77c1a6e2a9d2d299f113bb1e4"} Jul 08 18:53:59.028073 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:59.027749 2565 scope.go:117] "RemoveContainer" containerID="1cdb7a2871db6a61bccedb2baf5e22f93481fa68f6d464a0538d89eaf714a830" Jul 08 18:53:59.050558 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:59.050523 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp"] Jul 08 18:53:59.063767 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:59.063743 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-200f0c-8cf77c44b-76kpp"] Jul 08 18:53:59.928071 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:53:59.928038 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f651a3-1ae2-42b7-8967-d7994f7dbd18" path="/var/lib/kubelet/pods/46f651a3-1ae2-42b7-8967-d7994f7dbd18/volumes" Jul 08 18:54:59.414624 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.414586 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb"] Jul 08 18:54:59.415592 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.415568 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f651a3-1ae2-42b7-8967-d7994f7dbd18" containerName="maas-api" Jul 08 18:54:59.415788 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.415774 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f651a3-1ae2-42b7-8967-d7994f7dbd18" containerName="maas-api" Jul 08 18:54:59.416064 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.416040 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f651a3-1ae2-42b7-8967-d7994f7dbd18" containerName="maas-api" Jul 08 18:54:59.419326 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.419309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:54:59.422245 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.422224 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bd7541-serving-cert\"" Jul 08 18:54:59.429099 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.429077 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb"] Jul 08 18:54:59.578130 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.578098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:54:59.578275 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.578137 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx8xh\" (UniqueName: \"kubernetes.io/projected/63472c4d-654a-4e02-b696-9fef9d17da86-kube-api-access-wx8xh\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:54:59.578275 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.578187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63472c4d-654a-4e02-b696-9fef9d17da86-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:54:59.679265 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.679190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:54:59.679265 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.679231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx8xh\" (UniqueName: \"kubernetes.io/projected/63472c4d-654a-4e02-b696-9fef9d17da86-kube-api-access-wx8xh\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:54:59.679265 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.679263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63472c4d-654a-4e02-b696-9fef9d17da86-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:54:59.679487 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:54:59.679328 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-serving-cert: secret "maas-api-e2e-api-b-bd7541-serving-cert" not found Jul 08 18:54:59.679487 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:54:59.679394 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls podName:63472c4d-654a-4e02-b696-9fef9d17da86 nodeName:}" failed. No retries permitted until 2026-07-08 18:55:00.179378349 +0000 UTC m=+2234.802965411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls") pod "maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" (UID: "63472c4d-654a-4e02-b696-9fef9d17da86") : secret "maas-api-e2e-api-b-bd7541-serving-cert" not found Jul 08 18:54:59.680090 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.680068 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63472c4d-654a-4e02-b696-9fef9d17da86-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:54:59.690309 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:54:59.690287 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx8xh\" (UniqueName: \"kubernetes.io/projected/63472c4d-654a-4e02-b696-9fef9d17da86-kube-api-access-wx8xh\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:55:00.183326 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:00.183293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:55:00.185730 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:00.185714 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls\") pod \"maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:55:00.336199 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:00.336170 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:55:00.477018 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:00.476985 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb"] Jul 08 18:55:00.478439 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:55:00.478409 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63472c4d_654a_4e02_b696_9fef9d17da86.slice/crio-5f0d6212fe05f935fbdb9d9ac47309744748bf56f2f2772e8663767f9b4f0bca WatchSource:0}: Error finding container 5f0d6212fe05f935fbdb9d9ac47309744748bf56f2f2772e8663767f9b4f0bca: Status 404 returned error can't find the container with id 5f0d6212fe05f935fbdb9d9ac47309744748bf56f2f2772e8663767f9b4f0bca Jul 08 18:55:01.408616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:01.408531 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" event={"ID":"63472c4d-654a-4e02-b696-9fef9d17da86","Type":"ContainerStarted","Data":"83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5"} Jul 08 18:55:01.408616 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:01.408597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" event={"ID":"63472c4d-654a-4e02-b696-9fef9d17da86","Type":"ContainerStarted","Data":"5f0d6212fe05f935fbdb9d9ac47309744748bf56f2f2772e8663767f9b4f0bca"} Jul 08 18:55:01.408829 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:01.408752 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:55:01.426901 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:01.426860 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" podStartSLOduration=1.764323227 podStartE2EDuration="2.42684582s" totalImagesPullingTime="662.522593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:55:01.42560659 +0000 UTC m=+2236.049193671" watchObservedRunningTime="2026-07-08 18:55:01.42684582 +0000 UTC m=+2236.050432941" Jul 08 18:55:07.421097 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:07.421069 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:55:19.946606 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:19.946573 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb"] Jul 08 18:55:19.947185 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:19.946759 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" podUID="63472c4d-654a-4e02-b696-9fef9d17da86" containerName="maas-api" containerID="cri-o://83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5" gracePeriod=30 Jul 08 18:55:20.206022 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.205969 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:55:20.249379 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.249352 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63472c4d-654a-4e02-b696-9fef9d17da86-openshift-service-ca\") pod \"63472c4d-654a-4e02-b696-9fef9d17da86\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " Jul 08 18:55:20.249508 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.249439 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx8xh\" (UniqueName: \"kubernetes.io/projected/63472c4d-654a-4e02-b696-9fef9d17da86-kube-api-access-wx8xh\") pod \"63472c4d-654a-4e02-b696-9fef9d17da86\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " Jul 08 18:55:20.249556 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.249514 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls\") pod \"63472c4d-654a-4e02-b696-9fef9d17da86\" (UID: \"63472c4d-654a-4e02-b696-9fef9d17da86\") " Jul 08 18:55:20.249779 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.249747 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63472c4d-654a-4e02-b696-9fef9d17da86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63472c4d-654a-4e02-b696-9fef9d17da86" (UID: "63472c4d-654a-4e02-b696-9fef9d17da86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:55:20.249922 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.249854 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63472c4d-654a-4e02-b696-9fef9d17da86-openshift-service-ca\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:55:20.251573 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.251545 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63472c4d-654a-4e02-b696-9fef9d17da86-kube-api-access-wx8xh" (OuterVolumeSpecName: "kube-api-access-wx8xh") pod "63472c4d-654a-4e02-b696-9fef9d17da86" (UID: "63472c4d-654a-4e02-b696-9fef9d17da86"). InnerVolumeSpecName "kube-api-access-wx8xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:55:20.251682 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.251616 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63472c4d-654a-4e02-b696-9fef9d17da86" (UID: "63472c4d-654a-4e02-b696-9fef9d17da86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:55:20.350522 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.350501 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx8xh\" (UniqueName: \"kubernetes.io/projected/63472c4d-654a-4e02-b696-9fef9d17da86-kube-api-access-wx8xh\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:55:20.350522 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.350521 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63472c4d-654a-4e02-b696-9fef9d17da86-maas-api-tls\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:55:20.522552 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.522525 2565 generic.go:356] "Generic (PLEG): container finished" podID="63472c4d-654a-4e02-b696-9fef9d17da86" containerID="83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5" exitCode=0 Jul 08 18:55:20.522708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.522597 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" Jul 08 18:55:20.522708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.522615 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" event={"ID":"63472c4d-654a-4e02-b696-9fef9d17da86","Type":"ContainerDied","Data":"83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5"} Jul 08 18:55:20.522843 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.522712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb" event={"ID":"63472c4d-654a-4e02-b696-9fef9d17da86","Type":"ContainerDied","Data":"5f0d6212fe05f935fbdb9d9ac47309744748bf56f2f2772e8663767f9b4f0bca"} Jul 08 18:55:20.522843 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.522747 2565 scope.go:117] "RemoveContainer" containerID="83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5" Jul 08 18:55:20.533597 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.533572 2565 scope.go:117] "RemoveContainer" containerID="83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5" Jul 08 18:55:20.533867 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:55:20.533844 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5\": container with ID starting with 83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5 not found: ID does not exist" containerID="83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5" Jul 08 18:55:20.533944 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.533873 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5"} err="failed to get container status \"83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5\": rpc error: code = NotFound desc = could not find container \"83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5\": container with ID starting with 83ed4d1818f581da79d27b5350874a452141937b4abcf8658ee54b3fa65436e5 not found: ID does not exist" Jul 08 18:55:20.544827 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.544804 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb"] Jul 08 18:55:20.547307 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:20.547285 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd7541-7bfb4967d8-lj8bb"] Jul 08 18:55:21.928757 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:21.928719 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63472c4d-654a-4e02-b696-9fef9d17da86" path="/var/lib/kubelet/pods/63472c4d-654a-4e02-b696-9fef9d17da86/volumes" Jul 08 18:55:29.611539 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.611502 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v"] Jul 08 18:55:29.612242 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.612222 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63472c4d-654a-4e02-b696-9fef9d17da86" containerName="maas-api" Jul 08 18:55:29.612318 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.612246 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63472c4d-654a-4e02-b696-9fef9d17da86" containerName="maas-api" Jul 08 18:55:29.612354 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.612345 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63472c4d-654a-4e02-b696-9fef9d17da86" containerName="maas-api" Jul 08 18:55:29.616129 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.616109 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:29.620738 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.620711 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-956cea-serving-cert\"" Jul 08 18:55:29.638358 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.638325 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v"] Jul 08 18:55:29.727345 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.727317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:29.727508 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.727367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8c1654-90ab-40e3-84b5-80299aeb22ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:29.727508 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.727408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgnb\" (UniqueName: \"kubernetes.io/projected/ef8c1654-90ab-40e3-84b5-80299aeb22ad-kube-api-access-nlgnb\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:29.828463 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.828425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgnb\" (UniqueName: \"kubernetes.io/projected/ef8c1654-90ab-40e3-84b5-80299aeb22ad-kube-api-access-nlgnb\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:29.828589 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.828511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:29.828589 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.828550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8c1654-90ab-40e3-84b5-80299aeb22ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:29.828755 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:55:29.828655 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-serving-cert: secret "maas-api-e2e-shared-a-956cea-serving-cert" not found Jul 08 18:55:29.828755 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:55:29.828750 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls podName:ef8c1654-90ab-40e3-84b5-80299aeb22ad nodeName:}" failed. No retries permitted until 2026-07-08 18:55:30.328730613 +0000 UTC m=+2264.952317682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls") pod "maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" (UID: "ef8c1654-90ab-40e3-84b5-80299aeb22ad") : secret "maas-api-e2e-shared-a-956cea-serving-cert" not found Jul 08 18:55:29.829208 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.829192 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8c1654-90ab-40e3-84b5-80299aeb22ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:29.837833 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:29.837812 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgnb\" (UniqueName: \"kubernetes.io/projected/ef8c1654-90ab-40e3-84b5-80299aeb22ad-kube-api-access-nlgnb\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:30.334728 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:30.334693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:30.337322 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:30.337299 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-956cea-777ddf56-rsw4v\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:30.527090 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:30.527056 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:30.677564 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:30.677540 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v"] Jul 08 18:55:30.678645 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:55:30.678617 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8c1654_90ab_40e3_84b5_80299aeb22ad.slice/crio-32fe9084f72343a29a859526f50ff90c6acc98170846882a999db095f7681b0f WatchSource:0}: Error finding container 32fe9084f72343a29a859526f50ff90c6acc98170846882a999db095f7681b0f: Status 404 returned error can't find the container with id 32fe9084f72343a29a859526f50ff90c6acc98170846882a999db095f7681b0f Jul 08 18:55:31.592838 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:31.592804 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" event={"ID":"ef8c1654-90ab-40e3-84b5-80299aeb22ad","Type":"ContainerStarted","Data":"7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b"} Jul 08 18:55:31.593021 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:31.592855 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" event={"ID":"ef8c1654-90ab-40e3-84b5-80299aeb22ad","Type":"ContainerStarted","Data":"32fe9084f72343a29a859526f50ff90c6acc98170846882a999db095f7681b0f"} Jul 08 18:55:31.593021 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:31.592934 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:31.611508 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:31.611464 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" podStartSLOduration=2.20066141 podStartE2EDuration="2.611452238s" totalImagesPullingTime="410.790828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:55:31.61019217 +0000 UTC m=+2266.233779252" watchObservedRunningTime="2026-07-08 18:55:31.611452238 +0000 UTC m=+2266.235039318" Jul 08 18:55:37.607824 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:37.607794 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:55:37.769330 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:37.769296 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf"] Jul 08 18:55:37.782401 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:37.782372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:37.785352 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:37.785325 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf"] Jul 08 18:55:37.785547 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:37.785361 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c926a5-serving-cert\"" Jul 08 18:55:37.901063 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:37.900998 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0745c354-7d06-4e6f-ab4c-2791557ec369-maas-api-tls\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:37.901063 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:37.901031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0745c354-7d06-4e6f-ab4c-2791557ec369-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:37.901213 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:37.901105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4ckv\" (UniqueName: \"kubernetes.io/projected/0745c354-7d06-4e6f-ab4c-2791557ec369-kube-api-access-k4ckv\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:38.002082 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.002057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0745c354-7d06-4e6f-ab4c-2791557ec369-maas-api-tls\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:38.002212 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.002090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0745c354-7d06-4e6f-ab4c-2791557ec369-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:38.002212 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.002150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ckv\" (UniqueName: \"kubernetes.io/projected/0745c354-7d06-4e6f-ab4c-2791557ec369-kube-api-access-k4ckv\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:38.002896 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.002876 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0745c354-7d06-4e6f-ab4c-2791557ec369-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:38.004527 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.004498 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0745c354-7d06-4e6f-ab4c-2791557ec369-maas-api-tls\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:38.010374 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.010352 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4ckv\" (UniqueName: \"kubernetes.io/projected/0745c354-7d06-4e6f-ab4c-2791557ec369-kube-api-access-k4ckv\") pod \"maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:38.096774 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.096755 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:38.243299 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.243271 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf"] Jul 08 18:55:38.244899 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:55:38.244876 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0745c354_7d06_4e6f_ab4c_2791557ec369.slice/crio-869d210d2867e0c733b8522b86a4634dda3a7d9127acf54fb96e4afeda364061 WatchSource:0}: Error finding container 869d210d2867e0c733b8522b86a4634dda3a7d9127acf54fb96e4afeda364061: Status 404 returned error can't find the container with id 869d210d2867e0c733b8522b86a4634dda3a7d9127acf54fb96e4afeda364061 Jul 08 18:55:38.643204 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:38.643168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" event={"ID":"0745c354-7d06-4e6f-ab4c-2791557ec369","Type":"ContainerStarted","Data":"869d210d2867e0c733b8522b86a4634dda3a7d9127acf54fb96e4afeda364061"} Jul 08 18:55:39.650391 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:39.650356 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" event={"ID":"0745c354-7d06-4e6f-ab4c-2791557ec369","Type":"ContainerStarted","Data":"294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b"} Jul 08 18:55:39.650841 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:39.650430 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:55:39.669949 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:39.669903 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" podStartSLOduration=1.720284052 podStartE2EDuration="2.66988801s" totalImagesPullingTime="949.603958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:55:39.666718654 +0000 UTC m=+2274.290305733" watchObservedRunningTime="2026-07-08 18:55:39.66988801 +0000 UTC m=+2274.293475090" Jul 08 18:55:45.661605 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:55:45.661580 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:57:46.018700 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:57:46.018652 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:57:46.021248 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:57:46.020745 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:57:46.028339 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:57:46.028319 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:57:46.030222 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:57:46.030207 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ggfnv_5926e59d-0612-4f42-bc6c-d45f60b419c3/ovn-acl-logging/0.log" Jul 08 18:58:47.898762 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:47.898727 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v"] Jul 08 18:58:47.899377 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:47.899201 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" podUID="ef8c1654-90ab-40e3-84b5-80299aeb22ad" containerName="maas-api" containerID="cri-o://7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b" gracePeriod=30 Jul 08 18:58:48.182330 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.182304 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:58:48.301015 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.300981 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls\") pod \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " Jul 08 18:58:48.301202 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.301088 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8c1654-90ab-40e3-84b5-80299aeb22ad-openshift-service-ca\") pod \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " Jul 08 18:58:48.301202 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.301156 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlgnb\" (UniqueName: \"kubernetes.io/projected/ef8c1654-90ab-40e3-84b5-80299aeb22ad-kube-api-access-nlgnb\") pod \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\" (UID: \"ef8c1654-90ab-40e3-84b5-80299aeb22ad\") " Jul 08 18:58:48.301468 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.301439 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8c1654-90ab-40e3-84b5-80299aeb22ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef8c1654-90ab-40e3-84b5-80299aeb22ad" (UID: "ef8c1654-90ab-40e3-84b5-80299aeb22ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:58:48.303146 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.303116 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef8c1654-90ab-40e3-84b5-80299aeb22ad" (UID: "ef8c1654-90ab-40e3-84b5-80299aeb22ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:58:48.303253 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.303154 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8c1654-90ab-40e3-84b5-80299aeb22ad-kube-api-access-nlgnb" (OuterVolumeSpecName: "kube-api-access-nlgnb") pod "ef8c1654-90ab-40e3-84b5-80299aeb22ad" (UID: "ef8c1654-90ab-40e3-84b5-80299aeb22ad"). InnerVolumeSpecName "kube-api-access-nlgnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:58:48.402048 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.402017 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8c1654-90ab-40e3-84b5-80299aeb22ad-maas-api-tls\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:58:48.402048 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.402041 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8c1654-90ab-40e3-84b5-80299aeb22ad-openshift-service-ca\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:58:48.402048 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.402051 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlgnb\" (UniqueName: \"kubernetes.io/projected/ef8c1654-90ab-40e3-84b5-80299aeb22ad-kube-api-access-nlgnb\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:58:48.776993 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.776958 2565 generic.go:356] "Generic (PLEG): container finished" podID="ef8c1654-90ab-40e3-84b5-80299aeb22ad" containerID="7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b" exitCode=0 Jul 08 18:58:48.777175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.777012 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" Jul 08 18:58:48.777175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.777029 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" event={"ID":"ef8c1654-90ab-40e3-84b5-80299aeb22ad","Type":"ContainerDied","Data":"7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b"} Jul 08 18:58:48.777175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.777069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v" event={"ID":"ef8c1654-90ab-40e3-84b5-80299aeb22ad","Type":"ContainerDied","Data":"32fe9084f72343a29a859526f50ff90c6acc98170846882a999db095f7681b0f"} Jul 08 18:58:48.777175 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.777096 2565 scope.go:117] "RemoveContainer" containerID="7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b" Jul 08 18:58:48.786919 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.786904 2565 scope.go:117] "RemoveContainer" containerID="7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b" Jul 08 18:58:48.787158 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:58:48.787143 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b\": container with ID starting with 7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b not found: ID does not exist" containerID="7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b" Jul 08 18:58:48.787198 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.787166 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b"} err="failed to get container status \"7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b\": rpc error: code = NotFound desc = could not find container \"7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b\": container with ID starting with 7aab62a0915dee53df9bd17d9b0a8eca1a19490df080c885df9c611dafa8b56b not found: ID does not exist" Jul 08 18:58:48.799202 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.799166 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v"] Jul 08 18:58:48.803460 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:48.803443 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-956cea-777ddf56-rsw4v"] Jul 08 18:58:49.927803 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:49.927775 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8c1654-90ab-40e3-84b5-80299aeb22ad" path="/var/lib/kubelet/pods/ef8c1654-90ab-40e3-84b5-80299aeb22ad/volumes" Jul 08 18:58:54.939020 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:54.938973 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf"] Jul 08 18:58:54.939487 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:54.939224 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" podUID="0745c354-7d06-4e6f-ab4c-2791557ec369" containerName="maas-api" containerID="cri-o://294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b" gracePeriod=30 Jul 08 18:58:55.200645 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.200591 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:58:55.261995 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.261973 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4ckv\" (UniqueName: \"kubernetes.io/projected/0745c354-7d06-4e6f-ab4c-2791557ec369-kube-api-access-k4ckv\") pod \"0745c354-7d06-4e6f-ab4c-2791557ec369\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " Jul 08 18:58:55.262143 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.262074 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0745c354-7d06-4e6f-ab4c-2791557ec369-maas-api-tls\") pod \"0745c354-7d06-4e6f-ab4c-2791557ec369\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " Jul 08 18:58:55.262143 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.262121 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0745c354-7d06-4e6f-ab4c-2791557ec369-openshift-service-ca\") pod \"0745c354-7d06-4e6f-ab4c-2791557ec369\" (UID: \"0745c354-7d06-4e6f-ab4c-2791557ec369\") " Jul 08 18:58:55.262577 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.262548 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0745c354-7d06-4e6f-ab4c-2791557ec369-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0745c354-7d06-4e6f-ab4c-2791557ec369" (UID: "0745c354-7d06-4e6f-ab4c-2791557ec369"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:58:55.264097 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.264073 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0745c354-7d06-4e6f-ab4c-2791557ec369-kube-api-access-k4ckv" (OuterVolumeSpecName: "kube-api-access-k4ckv") pod "0745c354-7d06-4e6f-ab4c-2791557ec369" (UID: "0745c354-7d06-4e6f-ab4c-2791557ec369"). InnerVolumeSpecName "kube-api-access-k4ckv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:58:55.264265 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.264242 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0745c354-7d06-4e6f-ab4c-2791557ec369-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0745c354-7d06-4e6f-ab4c-2791557ec369" (UID: "0745c354-7d06-4e6f-ab4c-2791557ec369"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:58:55.362831 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.362806 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0745c354-7d06-4e6f-ab4c-2791557ec369-maas-api-tls\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:58:55.362831 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.362829 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0745c354-7d06-4e6f-ab4c-2791557ec369-openshift-service-ca\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:58:55.362999 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.362839 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4ckv\" (UniqueName: \"kubernetes.io/projected/0745c354-7d06-4e6f-ab4c-2791557ec369-kube-api-access-k4ckv\") on node \"ip-10-0-136-178.ec2.internal\" DevicePath \"\"" Jul 08 18:58:55.827303 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.827272 2565 generic.go:356] "Generic (PLEG): container finished" podID="0745c354-7d06-4e6f-ab4c-2791557ec369" containerID="294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b" exitCode=0 Jul 08 18:58:55.827478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.827317 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" Jul 08 18:58:55.827478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.827359 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" event={"ID":"0745c354-7d06-4e6f-ab4c-2791557ec369","Type":"ContainerDied","Data":"294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b"} Jul 08 18:58:55.827478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.827419 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf" event={"ID":"0745c354-7d06-4e6f-ab4c-2791557ec369","Type":"ContainerDied","Data":"869d210d2867e0c733b8522b86a4634dda3a7d9127acf54fb96e4afeda364061"} Jul 08 18:58:55.827478 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.827448 2565 scope.go:117] "RemoveContainer" containerID="294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b" Jul 08 18:58:55.837236 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.837220 2565 scope.go:117] "RemoveContainer" containerID="294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b" Jul 08 18:58:55.837488 ip-10-0-136-178 kubenswrapper[2565]: E0708 18:58:55.837469 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b\": container with ID starting with 294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b not found: ID does not exist" containerID="294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b" Jul 08 18:58:55.837560 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.837501 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b"} err="failed to get container status \"294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b\": rpc error: code = NotFound desc = could not find container \"294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b\": container with ID starting with 294e34165650feae0e1cab7b6b7f1ea3a02eb107a3f03946b2b368074c9e3d4b not found: ID does not exist" Jul 08 18:58:55.852448 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.852428 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf"] Jul 08 18:58:55.854923 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.854906 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c926a5-cd947b554-hl9sf"] Jul 08 18:58:55.928552 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:58:55.928524 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0745c354-7d06-4e6f-ab4c-2791557ec369" path="/var/lib/kubelet/pods/0745c354-7d06-4e6f-ab4c-2791557ec369/volumes" Jul 08 18:59:06.415360 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:06.415326 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-c8fwx_7438f9bd-2911-460e-afe2-e2f4abfc2b00/manager/0.log" Jul 08 18:59:06.538632 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:06.538605 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vjldn_ac29eed7-6326-42d7-8e18-26e56809718e/manager/0.log" Jul 08 18:59:06.660480 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:06.660451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787785b8d-27vss_a518d7dc-8080-4d99-8ec9-44b77e901b91/manager/0.log" Jul 08 18:59:06.783154 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:06.783135 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x59v7_9d992c01-aead-4782-8b0e-df526cf1453f/server/0.log" Jul 08 18:59:06.904883 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:06.904863 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-tlmk4_2dd4082d-29d8-495e-8774-fe298a1c6a4b/manager/0.log" Jul 08 18:59:07.155341 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:07.155274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578597fb9c-87mtb_ee24932a-c92e-4f69-83de-8de67c558069/copy-manifests/0.log" Jul 08 18:59:07.168528 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:07.168504 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578597fb9c-87mtb_ee24932a-c92e-4f69-83de-8de67c558069/manager/0.log" Jul 08 18:59:08.126615 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.126591 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp_52021da0-e11d-41eb-846a-f1074a6639c4/util/0.log" Jul 08 18:59:08.132708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.132656 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp_52021da0-e11d-41eb-846a-f1074a6639c4/pull/0.log" Jul 08 18:59:08.138028 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.138011 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp_52021da0-e11d-41eb-846a-f1074a6639c4/extract/0.log" Jul 08 18:59:08.263357 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.263335 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj_2db70365-7b9e-4e32-9c87-05bf0c316ccb/pull/0.log" Jul 08 18:59:08.269256 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.269230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj_2db70365-7b9e-4e32-9c87-05bf0c316ccb/extract/0.log" Jul 08 18:59:08.274531 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.274514 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj_2db70365-7b9e-4e32-9c87-05bf0c316ccb/util/0.log" Jul 08 18:59:08.396476 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.396417 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p_57886b32-8a1a-4cce-8729-dff55986dcca/util/0.log" Jul 08 18:59:08.402351 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.402331 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p_57886b32-8a1a-4cce-8729-dff55986dcca/pull/0.log" Jul 08 18:59:08.408193 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.408173 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p_57886b32-8a1a-4cce-8729-dff55986dcca/extract/0.log" Jul 08 18:59:08.525381 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.525358 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j_2f2007a1-b0d4-4579-9875-c29531b12604/util/0.log" Jul 08 18:59:08.532132 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.532115 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j_2f2007a1-b0d4-4579-9875-c29531b12604/pull/0.log" Jul 08 18:59:08.537938 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.537920 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j_2f2007a1-b0d4-4579-9875-c29531b12604/extract/0.log" Jul 08 18:59:08.796294 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.796271 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bqmq2_266baead-51c9-445c-9a27-292512147c66/manager/0.log" Jul 08 18:59:08.913738 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:08.913719 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s69ms_b1fcac96-320f-407d-9461-b007c5d314fd/manager/0.log" Jul 08 18:59:09.182548 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:09.182481 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-mx9zk_f9041d2e-f065-4c18-b340-0b2b47779f41/registry-server/0.log" Jul 08 18:59:09.308023 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:09.307996 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k8pcp_ce443991-19b1-4404-ac73-de7c475f248b/manager/0.log" Jul 08 18:59:09.559669 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:09.559632 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-557r4_d72084c8-cce0-4f8f-8de5-868e03bf2900/manager/0.log" Jul 08 18:59:10.300966 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:10.300939 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb94956b8-s895h_c2957457-c64b-4f9a-b886-7b08c4fa7902/kube-auth-proxy/0.log" Jul 08 18:59:10.420816 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:10.420791 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fbrl4_7857d7c4-d98d-4e46-a602-8f7d5535baad/istio-proxy/1.log" Jul 08 18:59:10.541648 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:10.541620 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-jrmnv_f7407027-91bb-4bea-892b-ff9f42621617/payload-pre-processing/0.log" Jul 08 18:59:10.664111 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:10.664083 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jkjzd_0001ac5d-2a97-428f-89e4-364bab76ffb2/payload-processing/0.log" Jul 08 18:59:11.141943 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:11.141916 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6qvq4_c592aefe-60c3-4bf2-9c3c-26c3e71676b4/main/0.log" Jul 08 18:59:11.486195 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:11.486172 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9bm7n_85654501-4f4c-4055-b389-56421de1500e/main/0.log" Jul 08 18:59:15.835089 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.835057 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6qtft/must-gather-kjc99"] Jul 08 18:59:15.835753 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.835732 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0745c354-7d06-4e6f-ab4c-2791557ec369" containerName="maas-api" Jul 08 18:59:15.835753 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.835754 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0745c354-7d06-4e6f-ab4c-2791557ec369" containerName="maas-api" Jul 08 18:59:15.835928 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.835775 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8c1654-90ab-40e3-84b5-80299aeb22ad" containerName="maas-api" Jul 08 18:59:15.835928 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.835782 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8c1654-90ab-40e3-84b5-80299aeb22ad" containerName="maas-api" Jul 08 18:59:15.836029 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.835987 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8c1654-90ab-40e3-84b5-80299aeb22ad" containerName="maas-api" Jul 08 18:59:15.836029 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.836004 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0745c354-7d06-4e6f-ab4c-2791557ec369" containerName="maas-api" Jul 08 18:59:15.841141 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.841121 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qtft/must-gather-kjc99" Jul 08 18:59:15.843970 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.843943 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qtft\"/\"kube-root-ca.crt\"" Jul 08 18:59:15.844108 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.843973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6qtft\"/\"default-dockercfg-4257n\"" Jul 08 18:59:15.844108 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.843974 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qtft\"/\"openshift-service-ca.crt\"" Jul 08 18:59:15.848124 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.848100 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qtft/must-gather-kjc99"] Jul 08 18:59:15.928911 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.928882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrgdl\" (UniqueName: \"kubernetes.io/projected/cd7d272f-62bb-4752-a169-ea653f9aaa00-kube-api-access-nrgdl\") pod \"must-gather-kjc99\" (UID: \"cd7d272f-62bb-4752-a169-ea653f9aaa00\") " pod="openshift-must-gather-6qtft/must-gather-kjc99" Jul 08 18:59:15.929039 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:15.928968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd7d272f-62bb-4752-a169-ea653f9aaa00-must-gather-output\") pod \"must-gather-kjc99\" (UID: \"cd7d272f-62bb-4752-a169-ea653f9aaa00\") " pod="openshift-must-gather-6qtft/must-gather-kjc99" Jul 08 18:59:16.029922 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:16.029899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd7d272f-62bb-4752-a169-ea653f9aaa00-must-gather-output\") pod \"must-gather-kjc99\" (UID: \"cd7d272f-62bb-4752-a169-ea653f9aaa00\") " pod="openshift-must-gather-6qtft/must-gather-kjc99" Jul 08 18:59:16.030031 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:16.029960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrgdl\" (UniqueName: \"kubernetes.io/projected/cd7d272f-62bb-4752-a169-ea653f9aaa00-kube-api-access-nrgdl\") pod \"must-gather-kjc99\" (UID: \"cd7d272f-62bb-4752-a169-ea653f9aaa00\") " pod="openshift-must-gather-6qtft/must-gather-kjc99" Jul 08 18:59:16.030285 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:16.030267 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd7d272f-62bb-4752-a169-ea653f9aaa00-must-gather-output\") pod \"must-gather-kjc99\" (UID: \"cd7d272f-62bb-4752-a169-ea653f9aaa00\") " pod="openshift-must-gather-6qtft/must-gather-kjc99" Jul 08 18:59:16.038822 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:16.038804 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrgdl\" (UniqueName: \"kubernetes.io/projected/cd7d272f-62bb-4752-a169-ea653f9aaa00-kube-api-access-nrgdl\") pod \"must-gather-kjc99\" (UID: \"cd7d272f-62bb-4752-a169-ea653f9aaa00\") " pod="openshift-must-gather-6qtft/must-gather-kjc99" Jul 08 18:59:16.152051 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:16.151988 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qtft/must-gather-kjc99" Jul 08 18:59:16.299680 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:16.299638 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qtft/must-gather-kjc99"] Jul 08 18:59:16.299795 ip-10-0-136-178 kubenswrapper[2565]: W0708 18:59:16.299778 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd7d272f_62bb_4752_a169_ea653f9aaa00.slice/crio-22fc23b0c37d4d6a80a383cb8c283c78c745961f974e36226d3c098301c392b0 WatchSource:0}: Error finding container 22fc23b0c37d4d6a80a383cb8c283c78c745961f974e36226d3c098301c392b0: Status 404 returned error can't find the container with id 22fc23b0c37d4d6a80a383cb8c283c78c745961f974e36226d3c098301c392b0 Jul 08 18:59:16.301936 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:16.301920 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:59:16.956732 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:16.956693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qtft/must-gather-kjc99" event={"ID":"cd7d272f-62bb-4752-a169-ea653f9aaa00","Type":"ContainerStarted","Data":"22fc23b0c37d4d6a80a383cb8c283c78c745961f974e36226d3c098301c392b0"} Jul 08 18:59:17.966943 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:17.966901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qtft/must-gather-kjc99" event={"ID":"cd7d272f-62bb-4752-a169-ea653f9aaa00","Type":"ContainerStarted","Data":"54cf7207ec1515d794791ef7d3f96c1da72ec38266d77fffda84277a16b78d12"} Jul 08 18:59:17.967433 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:17.966968 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qtft/must-gather-kjc99" event={"ID":"cd7d272f-62bb-4752-a169-ea653f9aaa00","Type":"ContainerStarted","Data":"e07c63dbb6596874a3b0e5073dce533e62e44f33564013a2743897b0e69e07c8"} Jul 08 18:59:17.985409 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:17.985353 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6qtft/must-gather-kjc99" podStartSLOduration=2.043663094 podStartE2EDuration="2.985331883s" totalImagesPullingTime="941.668789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:17.983485203 +0000 UTC m=+2492.607072285" watchObservedRunningTime="2026-07-08 18:59:17.985331883 +0000 UTC m=+2492.608918967" Jul 08 18:59:18.787093 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:18.787063 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v7kp9_d38d4dce-6648-426f-98f7-d2fbaf4e89df/global-pull-secret-syncer/0.log" Jul 08 18:59:18.897932 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:18.897901 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vldb6_f8cf354c-96c9-4feb-a464-d48b6a8d7fdf/konnectivity-agent/0.log" Jul 08 18:59:19.000593 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:19.000559 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-178.ec2.internal_a4bc3b05febacfc431990abc062b9833/haproxy/0.log" Jul 08 18:59:23.126204 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.126172 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp_52021da0-e11d-41eb-846a-f1074a6639c4/extract/0.log" Jul 08 18:59:23.162208 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.162178 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp_52021da0-e11d-41eb-846a-f1074a6639c4/util/0.log" Jul 08 18:59:23.209830 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.209797 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wznmp_52021da0-e11d-41eb-846a-f1074a6639c4/pull/0.log" Jul 08 18:59:23.283047 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.283012 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj_2db70365-7b9e-4e32-9c87-05bf0c316ccb/extract/0.log" Jul 08 18:59:23.322436 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.322404 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj_2db70365-7b9e-4e32-9c87-05bf0c316ccb/util/0.log" Jul 08 18:59:23.358847 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.358818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07qcfj_2db70365-7b9e-4e32-9c87-05bf0c316ccb/pull/0.log" Jul 08 18:59:23.398230 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.398153 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p_57886b32-8a1a-4cce-8729-dff55986dcca/extract/0.log" Jul 08 18:59:23.426611 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.426579 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p_57886b32-8a1a-4cce-8729-dff55986dcca/util/0.log" Jul 08 18:59:23.462714 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.462654 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qst7p_57886b32-8a1a-4cce-8729-dff55986dcca/pull/0.log" Jul 08 18:59:23.502630 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.502594 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j_2f2007a1-b0d4-4579-9875-c29531b12604/extract/0.log" Jul 08 18:59:23.548798 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.548765 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j_2f2007a1-b0d4-4579-9875-c29531b12604/util/0.log" Jul 08 18:59:23.588233 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.588200 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17s64j_2f2007a1-b0d4-4579-9875-c29531b12604/pull/0.log" Jul 08 18:59:23.974836 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:23.974802 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bqmq2_266baead-51c9-445c-9a27-292512147c66/manager/0.log" Jul 08 18:59:24.024427 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:24.024394 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s69ms_b1fcac96-320f-407d-9461-b007c5d314fd/manager/0.log" Jul 08 18:59:24.125277 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:24.125228 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-mx9zk_f9041d2e-f065-4c18-b340-0b2b47779f41/registry-server/0.log" Jul 08 18:59:24.283353 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:24.283302 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k8pcp_ce443991-19b1-4404-ac73-de7c475f248b/manager/0.log" Jul 08 18:59:24.376694 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:24.376626 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-557r4_d72084c8-cce0-4f8f-8de5-868e03bf2900/manager/0.log" Jul 08 18:59:26.091981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.091903 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf4f22f0-1d09-4f37-80c5-bddd43e48f5d/alertmanager/0.log" Jul 08 18:59:26.119404 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.119364 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf4f22f0-1d09-4f37-80c5-bddd43e48f5d/config-reloader/0.log" Jul 08 18:59:26.144078 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.144023 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf4f22f0-1d09-4f37-80c5-bddd43e48f5d/kube-rbac-proxy-web/0.log" Jul 08 18:59:26.166160 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.166113 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf4f22f0-1d09-4f37-80c5-bddd43e48f5d/kube-rbac-proxy/0.log" Jul 08 18:59:26.196881 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.196849 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf4f22f0-1d09-4f37-80c5-bddd43e48f5d/kube-rbac-proxy-metric/0.log" Jul 08 18:59:26.221642 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.221617 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf4f22f0-1d09-4f37-80c5-bddd43e48f5d/prom-label-proxy/0.log" Jul 08 18:59:26.247258 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.247233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf4f22f0-1d09-4f37-80c5-bddd43e48f5d/init-config-reloader/0.log" Jul 08 18:59:26.433393 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.433315 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-9jk6f_9ec587e7-85e2-481e-9574-2984b442d150/monitoring-plugin/0.log" Jul 08 18:59:26.465501 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.465471 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dqj6_8652526a-d355-4d28-bbd3-b7a70947c333/node-exporter/0.log" Jul 08 18:59:26.488008 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.487978 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dqj6_8652526a-d355-4d28-bbd3-b7a70947c333/kube-rbac-proxy/0.log" Jul 08 18:59:26.510334 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.510195 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dqj6_8652526a-d355-4d28-bbd3-b7a70947c333/init-textfile/0.log" Jul 08 18:59:26.700122 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.700049 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-hnngf_5bafd724-094e-4ff1-9465-501558427f83/kube-rbac-proxy-main/0.log" Jul 08 18:59:26.721551 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.721523 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-hnngf_5bafd724-094e-4ff1-9465-501558427f83/kube-rbac-proxy-self/0.log" Jul 08 18:59:26.742791 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.742750 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-hnngf_5bafd724-094e-4ff1-9465-501558427f83/openshift-state-metrics/0.log" Jul 08 18:59:26.950433 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.950354 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-2std7_faa2d397-fe25-40ae-817b-6aabb6d678a1/prometheus-operator/0.log" Jul 08 18:59:26.981981 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:26.981953 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-2std7_faa2d397-fe25-40ae-817b-6aabb6d678a1/kube-rbac-proxy/0.log" Jul 08 18:59:27.006963 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:27.006895 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-tb8v8_e68efccd-724b-487e-89ac-e3dc8244e5f5/prometheus-operator-admission-webhook/0.log" Jul 08 18:59:28.375509 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.375480 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-xlmdv_a6e10b2d-d75a-41ba-993e-35bfe3217286/networking-console-plugin/0.log" Jul 08 18:59:28.711584 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.711499 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7"] Jul 08 18:59:28.718001 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.717972 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.731636 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.731604 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7"] Jul 08 18:59:28.884568 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.884514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dqf\" (UniqueName: \"kubernetes.io/projected/4d3432a3-ad2b-4372-8625-6052c053068d-kube-api-access-b7dqf\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.884922 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.884895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-proc\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.885239 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.885219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-podres\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.885445 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.885429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-sys\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.885655 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.885630 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-lib-modules\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.986613 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.986576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-lib-modules\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.986826 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.986692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dqf\" (UniqueName: \"kubernetes.io/projected/4d3432a3-ad2b-4372-8625-6052c053068d-kube-api-access-b7dqf\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.986826 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.986732 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-proc\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.986974 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.986831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-podres\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.986974 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.986876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-sys\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.987088 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.986993 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-sys\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.987144 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.987118 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-lib-modules\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.987490 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.987464 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-proc\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.987600 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.987577 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d3432a3-ad2b-4372-8625-6052c053068d-podres\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:28.996678 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:28.996632 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dqf\" (UniqueName: \"kubernetes.io/projected/4d3432a3-ad2b-4372-8625-6052c053068d-kube-api-access-b7dqf\") pod \"perf-node-gather-daemonset-hb4h7\" (UID: \"4d3432a3-ad2b-4372-8625-6052c053068d\") " pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:29.031122 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:29.030858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:29.151681 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:29.150477 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/5.log" Jul 08 18:59:29.154139 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:29.153251 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-g44vm_fac2c03c-dff8-4df8-8400-0e30266cabfa/console-operator/6.log" Jul 08 18:59:29.239548 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:29.239523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7"] Jul 08 18:59:29.676653 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:29.676584 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75ff4d5fd7-4bfg9_2c292b26-5946-4dd9-87fa-2c59b000e2ec/console/0.log" Jul 08 18:59:29.708393 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:29.708368 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-xzrfz_a1041e9a-201a-4897-83f8-afe6f64c3cf2/download-server/0.log" Jul 08 18:59:30.087474 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:30.086915 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" event={"ID":"4d3432a3-ad2b-4372-8625-6052c053068d","Type":"ContainerStarted","Data":"ccbf82875c0c154c9240f813681bd6efd1103b14aa43bce697a8abcd62a0b0ed"} Jul 08 18:59:30.087474 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:30.086986 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" event={"ID":"4d3432a3-ad2b-4372-8625-6052c053068d","Type":"ContainerStarted","Data":"3325af4fc3d1305cf151c88403ed698d6ae58898e5c6d92d27612db362b95020"} Jul 08 18:59:30.087474 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:30.087182 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:30.104976 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:30.104929 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" podStartSLOduration=2.104914089 podStartE2EDuration="2.104914089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:30.103315176 +0000 UTC m=+2504.726902248" watchObservedRunningTime="2026-07-08 18:59:30.104914089 +0000 UTC m=+2504.728501216" Jul 08 18:59:30.244869 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:30.244832 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-4dbqj_45e06774-ee41-48b8-9f76-420e92a9dd62/volume-data-source-validator/0.log" Jul 08 18:59:31.357765 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:31.357715 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dn82h_73787897-ffc8-47cb-befc-1d8af360d229/dns/0.log" Jul 08 18:59:31.378708 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:31.378679 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dn82h_73787897-ffc8-47cb-befc-1d8af360d229/kube-rbac-proxy/0.log" Jul 08 18:59:31.426990 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:31.426961 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c228s_d28adaf6-6f30-4afd-80cf-191bc1b23cc0/dns-node-resolver/0.log" Jul 08 18:59:31.967911 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:31.967885 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjk7q_9e769af3-83dc-4841-95a4-b514d5e6b179/node-ca/0.log" Jul 08 18:59:33.082948 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:33.082912 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb94956b8-s895h_c2957457-c64b-4f9a-b886-7b08c4fa7902/kube-auth-proxy/0.log" Jul 08 18:59:33.110041 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:33.110013 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fbrl4_7857d7c4-d98d-4e46-a602-8f7d5535baad/istio-proxy/0.log" Jul 08 18:59:33.147901 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:33.147879 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fbrl4_7857d7c4-d98d-4e46-a602-8f7d5535baad/istio-proxy/1.log" Jul 08 18:59:33.192041 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:33.191996 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-jrmnv_f7407027-91bb-4bea-892b-ff9f42621617/payload-pre-processing/0.log" Jul 08 18:59:33.250641 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:33.250610 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jkjzd_0001ac5d-2a97-428f-89e4-364bab76ffb2/payload-processing/0.log" Jul 08 18:59:33.935699 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:33.935649 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5d65v_e2ac3f0b-d6c5-4d56-ad27-e39d04727ed4/serve-healthcheck-canary/0.log" Jul 08 18:59:34.529847 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:34.529824 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ttw49_d4cac3c2-5442-4580-9d4c-9bf130fa8382/kube-rbac-proxy/0.log" Jul 08 18:59:34.551261 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:34.551223 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ttw49_d4cac3c2-5442-4580-9d4c-9bf130fa8382/exporter/0.log" Jul 08 18:59:34.574238 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:34.574214 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ttw49_d4cac3c2-5442-4580-9d4c-9bf130fa8382/extractor/0.log" Jul 08 18:59:36.108407 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:36.108376 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6qtft/perf-node-gather-daemonset-hb4h7" Jul 08 18:59:36.786003 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:36.785962 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-c8fwx_7438f9bd-2911-460e-afe2-e2f4abfc2b00/manager/0.log" Jul 08 18:59:36.920690 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:36.920624 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vjldn_ac29eed7-6326-42d7-8e18-26e56809718e/manager/0.log" Jul 08 18:59:37.034191 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:37.034162 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787785b8d-27vss_a518d7dc-8080-4d99-8ec9-44b77e901b91/manager/0.log" Jul 08 18:59:37.057580 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:37.057494 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x59v7_9d992c01-aead-4782-8b0e-df526cf1453f/server/0.log" Jul 08 18:59:37.139153 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:37.139114 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-tlmk4_2dd4082d-29d8-495e-8774-fe298a1c6a4b/manager/0.log" Jul 08 18:59:37.292532 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:37.292496 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578597fb9c-87mtb_ee24932a-c92e-4f69-83de-8de67c558069/manager/0.log" Jul 08 18:59:37.310956 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:37.310849 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578597fb9c-87mtb_ee24932a-c92e-4f69-83de-8de67c558069/copy-manifests/0.log" Jul 08 18:59:38.875548 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:38.875521 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9xhgj_0c9b2e84-9097-497a-b057-94e0d7cbb449/openshift-lws-operator/0.log" Jul 08 18:59:39.339843 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:39.339814 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb94956b8-s895h_c2957457-c64b-4f9a-b886-7b08c4fa7902/kube-auth-proxy/0.log" Jul 08 18:59:39.366114 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:39.366083 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fbrl4_7857d7c4-d98d-4e46-a602-8f7d5535baad/istio-proxy/0.log" Jul 08 18:59:39.400628 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:39.400602 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fbrl4_7857d7c4-d98d-4e46-a602-8f7d5535baad/istio-proxy/1.log" Jul 08 18:59:39.437910 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:39.437880 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-jrmnv_f7407027-91bb-4bea-892b-ff9f42621617/payload-pre-processing/0.log" Jul 08 18:59:39.483260 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:39.483218 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jkjzd_0001ac5d-2a97-428f-89e4-364bab76ffb2/payload-processing/0.log" Jul 08 18:59:40.742512 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:40.742472 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb94956b8-s895h_c2957457-c64b-4f9a-b886-7b08c4fa7902/kube-auth-proxy/0.log" Jul 08 18:59:40.764759 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:40.764730 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fbrl4_7857d7c4-d98d-4e46-a602-8f7d5535baad/istio-proxy/0.log" Jul 08 18:59:40.818507 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:40.818482 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fbrl4_7857d7c4-d98d-4e46-a602-8f7d5535baad/istio-proxy/1.log" Jul 08 18:59:40.853002 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:40.852978 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-jrmnv_f7407027-91bb-4bea-892b-ff9f42621617/payload-pre-processing/0.log" Jul 08 18:59:40.900831 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:40.900807 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jkjzd_0001ac5d-2a97-428f-89e4-364bab76ffb2/payload-processing/0.log" Jul 08 18:59:43.880450 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:43.880422 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-xvxln_7230e111-6e2c-4076-92b2-5121dd1999e1/migrator/0.log" Jul 08 18:59:43.901402 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:43.901378 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-xvxln_7230e111-6e2c-4076-92b2-5121dd1999e1/graceful-termination/0.log" Jul 08 18:59:44.294102 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:44.294075 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-4dpmn_38444fe4-2b88-467a-bd50-8183c175e6ed/kube-storage-version-migrator-operator/1.log" Jul 08 18:59:44.295058 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:44.295036 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-4dpmn_38444fe4-2b88-467a-bd50-8183c175e6ed/kube-storage-version-migrator-operator/0.log" Jul 08 18:59:45.374329 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.374305 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8lbkn_8e202106-9ec8-4675-8bd9-f3c3ba45cb3e/kube-multus/0.log" Jul 08 18:59:45.396481 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.396457 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzfqj_e7d23d0c-cd9d-43bf-b2bb-569e389eae4b/kube-multus-additional-cni-plugins/0.log" Jul 08 18:59:45.417609 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.417581 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzfqj_e7d23d0c-cd9d-43bf-b2bb-569e389eae4b/egress-router-binary-copy/0.log" Jul 08 18:59:45.437946 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.437925 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzfqj_e7d23d0c-cd9d-43bf-b2bb-569e389eae4b/cni-plugins/0.log" Jul 08 18:59:45.457621 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.457603 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzfqj_e7d23d0c-cd9d-43bf-b2bb-569e389eae4b/bond-cni-plugin/0.log" Jul 08 18:59:45.479635 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.479621 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzfqj_e7d23d0c-cd9d-43bf-b2bb-569e389eae4b/routeoverride-cni/0.log" Jul 08 18:59:45.499893 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.499877 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzfqj_e7d23d0c-cd9d-43bf-b2bb-569e389eae4b/whereabouts-cni-bincopy/0.log" Jul 08 18:59:45.520191 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.520175 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzfqj_e7d23d0c-cd9d-43bf-b2bb-569e389eae4b/whereabouts-cni/0.log" Jul 08 18:59:45.902162 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.902082 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4dl6h_79fe673d-c3eb-4eda-8558-6498fbd55bd7/network-metrics-daemon/0.log" Jul 08 18:59:45.920390 ip-10-0-136-178 kubenswrapper[2565]: I0708 18:59:45.920368 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4dl6h_79fe673d-c3eb-4eda-8558-6498fbd55bd7/kube-rbac-proxy/0.log"