Jul 16 15:35:44.086736 ip-10-0-128-48 systemd[1]: Starting Kubernetes Kubelet... Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.524347 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528565 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528578 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528583 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528587 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528590 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528593 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 15:35:44.596030 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528596 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528599 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528601 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528604 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528606 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528609 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528612 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528615 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528617 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528620 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528628 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528631 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528634 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528636 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528639 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528641 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528644 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528647 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528650 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528653 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 15:35:44.599628 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528655 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 15:35:44.599396 ip-10-0-128-48 systemd[1]: Started Kubernetes Kubelet. Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528658 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528660 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528663 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528666 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528668 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528671 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528674 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528676 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528679 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528681 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528684 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528687 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528689 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528692 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528694 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528697 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528699 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528702 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 15:35:44.600958 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528705 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528708 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528711 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528713 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528716 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528718 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528721 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528724 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528726 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528729 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528731 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528734 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528736 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528739 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528742 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528744 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528747 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528749 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528751 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528754 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 15:35:44.605559 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528756 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528759 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528761 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528764 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528766 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528769 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528772 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528774 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528777 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528779 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528782 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528784 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528789 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528793 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528797 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528800 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528803 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528806 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528808 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528811 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 15:35:44.606320 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528813 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528816 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528818 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528821 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528823 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528826 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528828 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528831 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528833 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528836 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528839 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528841 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528843 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.528846 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529521 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529528 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529531 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529534 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529536 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529539 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 15:35:44.612377 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529543 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529545 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529548 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529551 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529553 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529556 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529559 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529563 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529565 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529568 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529570 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529573 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529576 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529578 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529581 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529584 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529586 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529589 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529592 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529594 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 15:35:44.613797 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529597 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529599 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529601 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529604 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529606 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529609 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529612 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529614 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529617 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529619 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529622 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529625 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529628 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529630 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529633 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529635 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529640 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529643 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529647 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 15:35:44.615195 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529650 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529654 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529657 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529660 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529664 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529667 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529670 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529673 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529676 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529679 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529682 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529685 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529688 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529692 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529696 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529699 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529702 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529704 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529707 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 15:35:44.621137 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529710 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529712 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529715 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529717 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529720 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529722 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529727 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529730 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529733 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529735 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529738 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529741 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529743 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529747 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529749 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529752 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529755 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529757 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529760 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 15:35:44.622401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529763 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529765 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529768 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529771 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529773 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529776 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529778 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529781 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529783 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529785 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529788 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529790 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529793 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529795 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529797 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.529800 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530149 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530156 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530159 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530164 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 15:35:44.623215 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530167 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530170 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530172 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530175 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530177 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530180 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530182 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530185 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530188 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530191 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530193 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530196 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530198 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530201 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530203 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530206 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530208 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530211 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530213 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530216 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 15:35:44.625331 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530218 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530221 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530223 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530226 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530228 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530231 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530233 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530236 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530238 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530244 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530247 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530252 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530255 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530259 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530261 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530264 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530266 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530269 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530271 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 15:35:44.626067 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530274 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530277 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530279 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530282 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530284 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530286 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530289 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530291 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530294 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530296 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530299 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530301 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530304 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530306 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530308 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530311 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530314 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530316 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530319 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 15:35:44.714285 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530321 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530324 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530326 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530330 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530333 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530336 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530339 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530342 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530345 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530347 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530350 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530352 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530355 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530357 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530360 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530364 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530367 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530370 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530374 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 15:35:44.714921 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530377 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530379 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530382 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530385 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530387 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530391 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530394 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530398 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530401 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530404 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530406 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530409 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530412 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530414 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530417 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530419 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530423 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.530425 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530532 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530540 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530547 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 15:35:44.717754 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530551 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530555 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530559 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530564 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530570 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530574 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530577 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530580 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530584 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530587 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530589 2573 flags.go:64] FLAG: --cgroup-root="" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530592 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530595 2573 flags.go:64] FLAG: --client-ca-file="" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530598 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530601 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530605 2573 flags.go:64] FLAG: --cluster-domain="" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530616 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530620 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530623 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530627 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530631 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530634 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530637 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 15:35:44.718469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530640 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530643 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530646 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530649 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530654 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530656 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530663 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530666 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530669 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530672 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530675 2573 flags.go:64] FLAG: --enable-server="true" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530678 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530683 2573 flags.go:64] FLAG: --event-burst="100" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530686 2573 flags.go:64] FLAG: --event-qps="50" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530689 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530692 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530694 2573 flags.go:64] FLAG: --eviction-hard="" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530698 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530701 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530703 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530707 2573 flags.go:64] FLAG: --eviction-soft="" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530710 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530713 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530715 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530718 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 15:35:46.144379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530721 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530724 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530727 2573 flags.go:64] FLAG: --feature-gates="" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530730 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530733 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530736 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530739 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530742 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530745 2573 flags.go:64] FLAG: --help="false" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530747 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530750 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530754 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530757 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530761 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530765 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530767 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530770 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530773 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530776 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530779 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530782 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530785 2573 flags.go:64] FLAG: --kube-reserved="" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530787 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530791 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 15:35:46.145794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530794 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530796 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530799 2573 flags.go:64] FLAG: --lock-file="" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530802 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530804 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530807 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530812 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530815 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530818 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530821 2573 flags.go:64] FLAG: --logging-format="text" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530824 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530828 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530830 2573 flags.go:64] FLAG: --manifest-url="" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530833 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530837 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530840 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530844 2573 flags.go:64] FLAG: --max-pods="110" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530846 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530849 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530854 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530856 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530861 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530864 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530867 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530873 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 15:35:46.146509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530877 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530880 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530883 2573 flags.go:64] FLAG: --pod-cidr="" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530886 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530891 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530894 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530897 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530900 2573 flags.go:64] FLAG: --port="10250" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530903 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530906 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b63396e0cf01cc5c" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530909 2573 flags.go:64] FLAG: --qos-reserved="" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530912 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530915 2573 flags.go:64] FLAG: --register-node="true" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530918 2573 flags.go:64] FLAG: --register-with-taints="" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530921 2573 flags.go:64] FLAG: --registry-burst="10" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530924 2573 flags.go:64] FLAG: --registry-qps="5" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530927 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530930 2573 flags.go:64] FLAG: --reserved-memory="" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530933 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530936 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530939 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530942 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530944 2573 flags.go:64] FLAG: --runonce="false" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530947 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530950 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 15:35:46.148276 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530953 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530956 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530961 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530965 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530968 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530971 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530973 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530976 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530979 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530982 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530985 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530988 2573 flags.go:64] FLAG: --system-cgroups="" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530991 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530996 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.530999 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531001 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531005 2573 flags.go:64] FLAG: --tls-min-version="" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531008 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531011 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531013 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531016 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531019 2573 flags.go:64] FLAG: --v="2" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531023 2573 flags.go:64] FLAG: --version="false" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531026 2573 flags.go:64] FLAG: --vmodule="" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531030 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 15:35:46.148991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.531033 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531159 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531163 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531166 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531169 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531172 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531174 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531177 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531179 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531183 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531187 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531190 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531192 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531195 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531198 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531200 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531203 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531205 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531208 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531211 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 15:35:46.150072 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531214 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531217 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531219 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531222 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531224 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531227 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531230 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531232 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531235 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531237 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531240 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531242 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531244 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531247 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531249 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531252 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531256 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531259 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531262 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531265 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 15:35:46.150774 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531268 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531273 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531277 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531280 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531283 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531286 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531288 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531291 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531293 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531296 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531299 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531301 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531304 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531307 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531309 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531312 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531315 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531317 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531320 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531322 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 15:35:46.151846 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531325 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531327 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531330 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531332 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531335 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531337 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531340 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531342 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531344 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531347 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531350 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531352 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531355 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531359 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531363 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531365 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531369 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531373 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531376 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 15:35:46.153642 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531379 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531382 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531385 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531387 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531390 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531393 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531396 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531399 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531402 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531404 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531407 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531409 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531412 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531414 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531417 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531419 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531422 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531424 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531427 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 15:35:46.154147 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531430 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.531432 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.532140 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.539945 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.539964 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.539989 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.539997 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540081 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540089 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540093 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540097 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540101 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540106 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540111 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540115 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 15:35:46.154603 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540119 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540123 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540127 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540131 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540135 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540139 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540142 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540146 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540151 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540154 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540158 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540162 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540166 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540170 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540173 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540177 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540181 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540185 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540189 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540193 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 15:35:46.154993 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540198 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540214 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540219 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540224 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540228 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540232 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540236 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540241 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540245 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540249 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540253 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540257 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540261 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540265 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540268 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540272 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540276 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540280 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540283 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 15:35:46.155463 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540287 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540291 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540295 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540298 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540303 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540306 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540310 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540314 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540317 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540321 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540325 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540329 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540333 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540336 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540343 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540355 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540360 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540363 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540367 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540371 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 15:35:46.155927 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540375 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540379 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540383 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540387 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540391 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540394 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540398 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540402 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540406 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540412 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540416 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540420 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540424 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540428 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540432 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540435 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540440 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540443 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540462 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 15:35:46.156401 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540467 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540471 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540474 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540478 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540482 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540486 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540489 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540497 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540501 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540514 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540521 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540524 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540528 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.540536 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 15:35:46.156861 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540747 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540754 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540759 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540764 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540768 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540772 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540776 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540780 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540784 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540788 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540792 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540797 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540801 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540804 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540808 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540812 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540816 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540819 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540823 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540827 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 15:35:46.157209 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540833 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540837 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540841 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540845 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540849 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540853 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540859 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540863 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540874 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540878 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540882 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540886 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540890 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540894 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540898 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540902 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540906 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540910 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540914 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540917 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 15:35:46.157714 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540921 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540927 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540932 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540936 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540941 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540945 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540949 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540953 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540956 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540960 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540964 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540968 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540973 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540977 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540981 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540984 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540988 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540992 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.540997 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 15:35:46.158184 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541001 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541005 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541016 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541020 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541024 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541028 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541031 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541036 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541040 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541044 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541048 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541051 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541055 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541059 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541063 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541067 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541071 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541076 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541081 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541085 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 15:35:46.158650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541090 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541093 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541098 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541101 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541106 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541113 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541117 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541125 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541130 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541134 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541138 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541143 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541147 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541151 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541155 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541166 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541170 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541174 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541177 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 15:35:46.159146 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.541182 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.541189 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.541368 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.543971 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.545382 2573 server.go:1019] "Starting client certificate rotation" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.545481 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.545509 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.551700 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.554474 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.564689 2573 log.go:25] "Validated CRI v1 runtime API" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.568035 2573 log.go:25] "Validated CRI v1 image API" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.568858 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 15:35:46.159603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.571489 2573 fs.go:135] Filesystem UUIDs: map[264d5a81-0ff7-4faa-89d6-6a1e5df97090:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 92926ecf-5ee8-4d82-b7fd-a1bc996b5f64:/dev/nvme0n1p3] Jul 16 15:35:46.159919 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.571513 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 15:35:46.159949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.576202 2573 manager.go:216] Machine: {Timestamp:2026-07-16 15:35:44.575093624 +0000 UTC m=+0.347783612 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103275 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297ac46f4d09b00e4237770d57f738 SystemUUID:ec297ac4-6f4d-09b0-0e42-37770d57f738 BootID:bb80ea74-c6db-4277-96bf-895dc94152d5 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ac:da:e5:cb:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ac:da:e5:cb:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:a2:33:dc:f9:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 15:35:46.159949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.576298 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 15:35:46.159949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.576371 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 15:35:46.159949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.577751 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 15:35:46.159949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.579973 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 15:35:46.159949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.580002 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-48.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 15:35:46.159949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.580740 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.580753 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.580762 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.581550 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.583062 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586119 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586230 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586257 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586267 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586278 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586286 2573 kubelet.go:398] "Adding apiserver pod source" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586294 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586704 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.586729 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.589049 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.592340 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.592441 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.592819 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.592846 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.592856 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.592866 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.592878 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.592887 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.593571 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.593582 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.593604 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.593613 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.593628 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.593642 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.594556 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.594788 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.597446 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.597480 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.597500 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.598337 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bnkgv" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.598654 2573 server.go:1268] "Started kubelet" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.598717 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.598835 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.598970 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.600105 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 15:35:46.160161 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.600781 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.603789 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.606152 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.606292 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.606343 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.606833 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.606846 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.606848 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607041 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607050 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607333 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bnkgv" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607379 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607393 2573 factory.go:55] Registering systemd factory Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607403 2573 factory.go:223] Registration of the systemd container factory successfully Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607787 2573 factory.go:153] Registering CRI-O factory Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607800 2573 factory.go:223] Registration of the crio container factory successfully Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607820 2573 factory.go:103] Registering Raw factory Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.607853 2573 manager.go:1196] Started watching for new ooms in manager Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.608270 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.608508 2573 manager.go:349] Starting recovery of all containers Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.608336 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-48.ec2.internal.18c2cf215d51633f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-48.ec2.internal,UID:ip-10-0-128-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-48.ec2.internal,},FirstTimestamp:2026-07-16 15:35:44.598623039 +0000 UTC m=+0.371313028,LastTimestamp:2026-07-16 15:35:44.598623039 +0000 UTC m=+0.371313028,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-48.ec2.internal,}" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.609892 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:44.611951 2573 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/pids.max": read /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/pids.max: no such device Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.618723 2573 manager.go:354] Recovery completed Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.624551 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.626499 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientMemory" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.626525 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.626534 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientPID" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.627063 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.627073 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.627088 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.627075 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 15:35:46.161130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.627818 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientMemory" Jul 16 15:35:46.161953 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.627841 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 15:35:46.161953 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:44.627853 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientPID" Jul 16 15:35:46.161953 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.628741 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-48.ec2.internal.18c2cf215efaebd9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-48.ec2.internal,UID:ip-10-0-128-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-48.ec2.internal,},FirstTimestamp:2026-07-16 15:35:44.626510809 +0000 UTC m=+0.399200800,LastTimestamp:2026-07-16 15:35:44.626510809 +0000 UTC m=+0.399200800,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-48.ec2.internal,}" Jul 16 15:35:46.161953 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:44.706656 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.161953 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.141093 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.161953 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.149195 2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-48.ec2.internal\" not found" node="ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.199090 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.199076 2573 policy_none.go:49] "None policy: Start" Jul 16 15:35:46.199130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.199094 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 15:35:46.199130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.199105 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 15:35:46.200284 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.200273 2573 policy_none.go:47] "Start" Jul 16 15:35:46.229230 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.229215 2573 manager.go:337] "Starting Device Plugin manager" Jul 16 15:35:46.229312 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.229253 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 15:35:46.229312 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.229263 2573 server.go:87] "Starting device plugin registration server" Jul 16 15:35:46.229510 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.229494 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 15:35:46.229596 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.229511 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 15:35:46.229707 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.229688 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 15:35:46.229793 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.229779 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 15:35:46.229793 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.229793 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 15:35:46.230328 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.230304 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 15:35:46.230427 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.230363 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.329826 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.329802 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 15:35:46.331306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.330908 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientMemory" Jul 16 15:35:46.331306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.330935 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 15:35:46.331306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.330949 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientPID" Jul 16 15:35:46.331306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.330977 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.339967 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.339953 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.339967 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.339973 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-48.ec2.internal\": node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.364001 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.363985 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.375838 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.375819 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 15:35:46.377063 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.377044 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 15:35:46.377136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.377090 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 15:35:46.377136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.377110 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 15:35:46.377201 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.377140 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 15:35:46.379545 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.379528 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 15:35:46.464112 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.464067 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.478223 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.478203 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal"] Jul 16 15:35:46.478287 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.478277 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 15:35:46.479137 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.479116 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientMemory" Jul 16 15:35:46.479221 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.479145 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 15:35:46.479221 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.479161 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientPID" Jul 16 15:35:46.480659 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.480647 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 15:35:46.480775 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.480760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.480816 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.480809 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 15:35:46.481324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.481309 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientMemory" Jul 16 15:35:46.481386 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.481331 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 15:35:46.481386 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.481344 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientPID" Jul 16 15:35:46.481386 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.481366 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientMemory" Jul 16 15:35:46.481386 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.481386 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 15:35:46.481513 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.481399 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientPID" Jul 16 15:35:46.482506 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.482489 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.482590 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.482513 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 15:35:46.483136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.483121 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientMemory" Jul 16 15:35:46.483210 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.483150 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 15:35:46.483210 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.483164 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeHasSufficientPID" Jul 16 15:35:46.502136 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.502109 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-48.ec2.internal\" not found" node="ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.506303 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.506286 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-48.ec2.internal\" not found" node="ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.543624 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.543593 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b41c40da60a15333861b842213a29712-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal\" (UID: \"b41c40da60a15333861b842213a29712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.543691 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.543627 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b41c40da60a15333861b842213a29712-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal\" (UID: \"b41c40da60a15333861b842213a29712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.543691 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.543647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc266b970294ca7e6b02e58f7dfbfaf7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-48.ec2.internal\" (UID: \"fc266b970294ca7e6b02e58f7dfbfaf7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.564653 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.564634 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.644428 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.644407 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc266b970294ca7e6b02e58f7dfbfaf7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-48.ec2.internal\" (UID: \"fc266b970294ca7e6b02e58f7dfbfaf7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.644507 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.644434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b41c40da60a15333861b842213a29712-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal\" (UID: \"b41c40da60a15333861b842213a29712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.644507 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.644451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b41c40da60a15333861b842213a29712-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal\" (UID: \"b41c40da60a15333861b842213a29712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.644569 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.644507 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc266b970294ca7e6b02e58f7dfbfaf7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-48.ec2.internal\" (UID: \"fc266b970294ca7e6b02e58f7dfbfaf7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.644569 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.644513 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b41c40da60a15333861b842213a29712-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal\" (UID: \"b41c40da60a15333861b842213a29712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.644569 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.644537 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b41c40da60a15333861b842213a29712-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal\" (UID: \"b41c40da60a15333861b842213a29712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.665689 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.665672 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.766078 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.765998 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.804197 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.804174 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.809293 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.809278 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" Jul 16 15:35:46.866796 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.866771 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.878930 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.878911 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 15:35:46.879055 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.879039 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 15:35:46.946318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.946286 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 15:30:45 +0000 UTC" deadline="2028-03-02 05:10:48.149333489 +0000 UTC" Jul 16 15:35:46.946318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.946310 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14269h35m1.203026104s" Jul 16 15:35:46.967568 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:46.967548 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-48.ec2.internal\" not found" Jul 16 15:35:46.969133 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:46.969119 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 15:35:47.041349 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.041274 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" Jul 16 15:35:47.049465 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.049449 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 15:35:47.059644 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.059626 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 15:35:47.060818 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.060806 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" Jul 16 15:35:47.074273 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.074256 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 15:35:47.386405 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:47.386356 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc266b970294ca7e6b02e58f7dfbfaf7.slice/crio-34429859d677aacaab2f53e3d8fe52b7aa2a0052cad238edef49dd028a7ab339 WatchSource:0}: Error finding container 34429859d677aacaab2f53e3d8fe52b7aa2a0052cad238edef49dd028a7ab339: Status 404 returned error can't find the container with id 34429859d677aacaab2f53e3d8fe52b7aa2a0052cad238edef49dd028a7ab339 Jul 16 15:35:47.386865 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:47.386845 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41c40da60a15333861b842213a29712.slice/crio-a521fd1256d191c968ebda3c65635cff61fab174adfd700703da9400b20edc6c WatchSource:0}: Error finding container a521fd1256d191c968ebda3c65635cff61fab174adfd700703da9400b20edc6c: Status 404 returned error can't find the container with id a521fd1256d191c968ebda3c65635cff61fab174adfd700703da9400b20edc6c Jul 16 15:35:47.392570 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.392555 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 15:35:47.516831 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.516805 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 15:35:47.921324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.921296 2573 apiserver.go:52] "Watching apiserver" Jul 16 15:35:47.929346 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.929300 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 15:35:47.931119 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.931093 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-8k54x","openshift-network-operator/iptables-alerter-mshv5","openshift-ovn-kubernetes/ovnkube-node-hprmr","kube-system/konnectivity-agent-qxxr2","kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw","openshift-cluster-node-tuning-operator/tuned-pllg4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal","openshift-multus/multus-additional-cni-plugins-md42f","openshift-multus/multus-nfwrq","openshift-multus/network-metrics-daemon-nz489","kube-system/global-pull-secret-syncer-sdfct","openshift-dns/node-resolver-dh7cp","openshift-image-registry/node-ca-6s8bz"] Jul 16 15:35:47.932636 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.932603 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.934025 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.933960 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:47.934114 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:47.934077 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:35:47.935473 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.935451 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 15:35:47.935574 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.935507 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.935797 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.935776 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ld2nd\"" Jul 16 15:35:47.935953 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.935929 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 15:35:47.938114 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.937930 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gl22h\"" Jul 16 15:35:47.938672 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.938575 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:35:47.938904 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.938882 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 15:35:47.939243 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.939222 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 15:35:47.939335 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.939252 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 15:35:47.939335 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.939276 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 15:35:47.939335 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.939230 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 15:35:47.939501 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.939251 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 15:35:47.939968 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.939949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:47.940930 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.940903 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 15:35:47.940930 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.940918 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 15:35:47.941385 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.941367 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:47.942334 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.942317 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-scbd2\"" Jul 16 15:35:47.942580 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.942561 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n2f7p\"" Jul 16 15:35:47.942727 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.942565 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 15:35:47.942811 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.942778 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 15:35:47.942868 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.942712 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 15:35:47.943079 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.943062 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 15:35:47.943079 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.943063 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:47.944432 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.944415 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:47.944889 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.944871 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 15:35:47.945861 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.945844 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:47.947194 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.947098 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 15:35:47.947194 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.947138 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vkbkj\"" Jul 16 15:35:47.947716 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.947293 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 15:35:47.947716 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.947341 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 15:35:47.947716 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.947630 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nfwrq" Jul 16 15:35:47.947716 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.947684 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 15:35:47.947905 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.947725 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 15:35:47.948061 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.947905 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 15:35:47.948061 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.948038 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 15:35:47.948160 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.948112 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 15:35:47.948407 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.948382 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m2gxb\"" Jul 16 15:35:47.948580 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.948562 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 15:35:47.948765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.948734 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jbnh9\"" Jul 16 15:35:47.948941 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.948923 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8v9z6\"" Jul 16 15:35:47.949335 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.949278 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 15:35:47.949778 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.949722 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:47.949890 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:47.949792 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:35:47.950262 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.950242 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 15:35:47.951222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951202 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kmfgh\"" Jul 16 15:35:47.951309 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/065bbaff-e023-4be5-bd5b-be46329be9a3-agent-certs\") pod \"konnectivity-agent-qxxr2\" (UID: \"065bbaff-e023-4be5-bd5b-be46329be9a3\") " pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:35:47.951478 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/065bbaff-e023-4be5-bd5b-be46329be9a3-konnectivity-ca\") pod \"konnectivity-agent-qxxr2\" (UID: \"065bbaff-e023-4be5-bd5b-be46329be9a3\") " pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:35:47.951534 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951289 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 15:35:47.951534 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-sys-fs\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:47.951656 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-run\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.951656 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9x7p\" (UniqueName: \"kubernetes.io/projected/2308a825-c474-4f66-a4ca-12e024550bdf-kube-api-access-s9x7p\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.951656 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-device-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:47.951656 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysctl-d\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.951846 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:47.951846 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.951846 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951832 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952000 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951944 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysctl-conf\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952000 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.951981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-var-lib-kubelet\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-host\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952031 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 15:35:47.952095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61456613-eddd-4594-b4fa-c61c23d2976d-tmp\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlsv\" (UniqueName: \"kubernetes.io/projected/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-kube-api-access-xrlsv\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:47.952286 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysconfig\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952286 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-log-socket\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952286 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-serviceca\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:47.952286 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzns\" (UniqueName: \"kubernetes.io/projected/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-kube-api-access-cjzns\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:47.952286 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-lib-modules\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-systemd-units\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952329 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-slash\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-run-netns\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-etc-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-systemd\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-sys\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-kubelet\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2308a825-c474-4f66-a4ca-12e024550bdf-ovn-node-metrics-cert\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:47.952986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-registration-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:47.952986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-886zw\" (UniqueName: \"kubernetes.io/projected/61456613-eddd-4594-b4fa-c61c23d2976d-kube-api-access-886zw\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.952986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-var-lib-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-ovn\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-node-log\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952836 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-cni-bin\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.952986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.952978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-socket-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:47.953234 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:47.953234 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-modprobe-d\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.953234 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-kubernetes\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.953234 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-systemd\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.953415 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-ovnkube-config\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.953415 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-env-overrides\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.953415 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-host\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:47.953530 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61456613-eddd-4594-b4fa-c61c23d2976d-etc-tuned\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:47.953530 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqz9\" (UniqueName: \"kubernetes.io/projected/b8921e8d-2506-4ff1-abc3-21abb1038d4c-kube-api-access-vvqz9\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:47.953643 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:47.953643 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-cni-netd\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.953755 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.953666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-ovnkube-script-lib\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:47.953755 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:47.953696 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:35:47.956323 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.956302 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 15:35:47.976041 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.975862 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkmbv" Jul 16 15:35:47.983495 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:47.983457 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkmbv" Jul 16 15:35:48.043067 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.043049 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 15:35:48.054660 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-886zw\" (UniqueName: \"kubernetes.io/projected/61456613-eddd-4594-b4fa-c61c23d2976d-kube-api-access-886zw\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.054773 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-var-lib-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.054773 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-cni-bin\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.054773 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpx84\" (UniqueName: \"kubernetes.io/projected/01037690-454d-4b4a-8b72-c9ed93c0beaf-kube-api-access-hpx84\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.054773 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054755 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-var-lib-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.054968 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-socket-dir-parent\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.054968 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054811 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-cni-bin\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.054968 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-kubelet\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.054968 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.054968 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-modprobe-d\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.054968 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054949 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.054983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-kubernetes\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055005 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-modprobe-d\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055045 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-kubernetes\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72q4h\" (UniqueName: \"kubernetes.io/projected/76bc2888-9278-47da-bec1-0efc06a0aaca-kube-api-access-72q4h\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-multus-certs\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055116 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-host\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9295763d-a0e9-43f3-98c9-a86588ddf8f2-tmp-dir\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-system-cni-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055191 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-host\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:48.055217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/065bbaff-e023-4be5-bd5b-be46329be9a3-agent-certs\") pod \"konnectivity-agent-qxxr2\" (UID: \"065bbaff-e023-4be5-bd5b-be46329be9a3\") " pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-sys-fs\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-cnibin\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055296 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055297 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-sys-fs\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysctl-d\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055389 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysctl-d\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-os-release\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysctl-conf\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.055701 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-var-lib-kubelet\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-host\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055513 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlsv\" (UniqueName: \"kubernetes.io/projected/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-kube-api-access-xrlsv\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055536 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055542 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-var-lib-kubelet\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055545 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-host\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzns\" (UniqueName: \"kubernetes.io/projected/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-kube-api-access-cjzns\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysctl-conf\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-systemd-units\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-slash\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-etc-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055684 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-systemd-units\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055696 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-cnibin\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.056318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055734 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-slash\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-etc-openvswitch\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-sys\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-cni-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055811 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-sys\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-registration-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-ovn\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-node-log\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055914 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-registration-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055915 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055925 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-ovn\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-ovnkube-config\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055964 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-node-log\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.055999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056014 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76bc2888-9278-47da-bec1-0efc06a0aaca-iptables-alerter-script\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-socket-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-systemd\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-env-overrides\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76bc2888-9278-47da-bec1-0efc06a0aaca-host-slash\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056112 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7962491-319d-4d1c-8d70-74706e135117-kubelet-config\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1619a010-0b97-427e-9d72-5d0b576077ba-cni-binary-copy\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056139 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-run-systemd\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-netns\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61456613-eddd-4594-b4fa-c61c23d2976d-etc-tuned\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqz9\" (UniqueName: \"kubernetes.io/projected/b8921e8d-2506-4ff1-abc3-21abb1038d4c-kube-api-access-vvqz9\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:48.057675 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-cni-netd\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056258 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-socket-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-cni-netd\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-ovnkube-script-lib\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-os-release\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-conf-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-etc-kubernetes\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/065bbaff-e023-4be5-bd5b-be46329be9a3-konnectivity-ca\") pod \"konnectivity-agent-qxxr2\" (UID: \"065bbaff-e023-4be5-bd5b-be46329be9a3\") " pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-run\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9x7p\" (UniqueName: \"kubernetes.io/projected/2308a825-c474-4f66-a4ca-12e024550bdf-kube-api-access-s9x7p\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-system-cni-dir\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9295763d-a0e9-43f3-98c9-a86588ddf8f2-hosts-file\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1619a010-0b97-427e-9d72-5d0b576077ba-multus-daemon-config\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-device-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.058324 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-ovnkube-config\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056659 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-env-overrides\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056698 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-device-dir\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.056913 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056955 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2308a825-c474-4f66-a4ca-12e024550bdf-ovnkube-script-lib\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.056974 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs podName:b8921e8d-2506-4ff1-abc3-21abb1038d4c nodeName:}" failed. No retries permitted until 2026-07-16 15:35:48.556954537 +0000 UTC m=+2.995481359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs") pod "network-metrics-daemon-nz489" (UID: "b8921e8d-2506-4ff1-abc3-21abb1038d4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.056981 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-run\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblwx\" (UniqueName: \"kubernetes.io/projected/1619a010-0b97-427e-9d72-5d0b576077ba-kube-api-access-mblwx\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057043 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61456613-eddd-4594-b4fa-c61c23d2976d-tmp\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysconfig\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/065bbaff-e023-4be5-bd5b-be46329be9a3-konnectivity-ca\") pod \"konnectivity-agent-qxxr2\" (UID: \"065bbaff-e023-4be5-bd5b-be46329be9a3\") " pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-sysconfig\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.058977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-log-socket\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-k8s-cni-cncf-io\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-cni-bin\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057191 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-log-socket\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-serviceca\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-lib-modules\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-run-netns\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057360 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfwf\" (UniqueName: \"kubernetes.io/projected/9295763d-a0e9-43f3-98c9-a86588ddf8f2-kube-api-access-vjfwf\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7962491-319d-4d1c-8d70-74706e135117-dbus\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-lib-modules\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-run-netns\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-cni-multus\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-systemd\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-kubelet\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.059661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2308a825-c474-4f66-a4ca-12e024550bdf-ovn-node-metrics-cert\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.060320 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057653 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-serviceca\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:48.060320 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057681 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-hostroot\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.060320 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057718 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61456613-eddd-4594-b4fa-c61c23d2976d-etc-systemd\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.060320 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.057837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2308a825-c474-4f66-a4ca-12e024550bdf-host-kubelet\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.060320 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.059597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61456613-eddd-4594-b4fa-c61c23d2976d-tmp\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.060320 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.059609 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/065bbaff-e023-4be5-bd5b-be46329be9a3-agent-certs\") pod \"konnectivity-agent-qxxr2\" (UID: \"065bbaff-e023-4be5-bd5b-be46329be9a3\") " pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:35:48.060320 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.059835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61456613-eddd-4594-b4fa-c61c23d2976d-etc-tuned\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.060662 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.060386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2308a825-c474-4f66-a4ca-12e024550bdf-ovn-node-metrics-cert\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.067450 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.067425 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-886zw\" (UniqueName: \"kubernetes.io/projected/61456613-eddd-4594-b4fa-c61c23d2976d-kube-api-access-886zw\") pod \"tuned-pllg4\" (UID: \"61456613-eddd-4594-b4fa-c61c23d2976d\") " pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.068288 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.068242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlsv\" (UniqueName: \"kubernetes.io/projected/d94066e5-1ed5-4b3e-93d8-881ee4b3de4c-kube-api-access-xrlsv\") pod \"aws-ebs-csi-driver-node-r8ttw\" (UID: \"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.068801 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.068742 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzns\" (UniqueName: \"kubernetes.io/projected/cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4-kube-api-access-cjzns\") pod \"node-ca-6s8bz\" (UID: \"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4\") " pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:48.069150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.069131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9x7p\" (UniqueName: \"kubernetes.io/projected/2308a825-c474-4f66-a4ca-12e024550bdf-kube-api-access-s9x7p\") pod \"ovnkube-node-hprmr\" (UID: \"2308a825-c474-4f66-a4ca-12e024550bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.070688 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.070667 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqz9\" (UniqueName: \"kubernetes.io/projected/b8921e8d-2506-4ff1-abc3-21abb1038d4c-kube-api-access-vvqz9\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:48.157974 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.157942 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-hostroot\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.157986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpx84\" (UniqueName: \"kubernetes.io/projected/01037690-454d-4b4a-8b72-c9ed93c0beaf-kube-api-access-hpx84\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.158136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-socket-dir-parent\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-kubelet\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72q4h\" (UniqueName: \"kubernetes.io/projected/76bc2888-9278-47da-bec1-0efc06a0aaca-kube-api-access-72q4h\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.158136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-multus-certs\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9295763d-a0e9-43f3-98c9-a86588ddf8f2-tmp-dir\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.158136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-system-cni-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158136 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-cnibin\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158164 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-os-release\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-cnibin\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-cni-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76bc2888-9278-47da-bec1-0efc06a0aaca-iptables-alerter-script\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76bc2888-9278-47da-bec1-0efc06a0aaca-host-slash\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7962491-319d-4d1c-8d70-74706e135117-kubelet-config\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158401 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-cnibin\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1619a010-0b97-427e-9d72-5d0b576077ba-cni-binary-copy\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.158561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-netns\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-hostroot\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-os-release\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-conf-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-etc-kubernetes\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-socket-dir-parent\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-system-cni-dir\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9295763d-a0e9-43f3-98c9-a86588ddf8f2-hosts-file\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1619a010-0b97-427e-9d72-5d0b576077ba-multus-daemon-config\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblwx\" (UniqueName: \"kubernetes.io/projected/1619a010-0b97-427e-9d72-5d0b576077ba-kube-api-access-mblwx\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-k8s-cni-cncf-io\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-cni-bin\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfwf\" (UniqueName: \"kubernetes.io/projected/9295763d-a0e9-43f3-98c9-a86588ddf8f2-kube-api-access-vjfwf\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76bc2888-9278-47da-bec1-0efc06a0aaca-host-slash\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.159222 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7962491-319d-4d1c-8d70-74706e135117-dbus\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-cni-multus\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158905 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-cni-multus\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7962491-319d-4d1c-8d70-74706e135117-dbus\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.158984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-system-cni-dir\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159037 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9295763d-a0e9-43f3-98c9-a86588ddf8f2-hosts-file\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159125 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7962491-319d-4d1c-8d70-74706e135117-kubelet-config\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159141 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76bc2888-9278-47da-bec1-0efc06a0aaca-iptables-alerter-script\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159148 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159175 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-multus-certs\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-netns\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159147 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-cnibin\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159208 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-conf-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159211 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-multus-cni-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.159881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159244 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-kubelet\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-os-release\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159288 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-run-k8s-cni-cncf-io\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-etc-kubernetes\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159444 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01037690-454d-4b4a-8b72-c9ed93c0beaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9295763d-a0e9-43f3-98c9-a86588ddf8f2-tmp-dir\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.159517 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1619a010-0b97-427e-9d72-5d0b576077ba-cni-binary-copy\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.159576 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret podName:e7962491-319d-4d1c-8d70-74706e135117 nodeName:}" failed. No retries permitted until 2026-07-16 15:35:48.659560752 +0000 UTC m=+3.098087568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret") pod "global-pull-secret-syncer-sdfct" (UID: "e7962491-319d-4d1c-8d70-74706e135117") : object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159575 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-os-release\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1619a010-0b97-427e-9d72-5d0b576077ba-multus-daemon-config\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-system-cni-dir\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1619a010-0b97-427e-9d72-5d0b576077ba-host-var-lib-cni-bin\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.160497 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.159905 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01037690-454d-4b4a-8b72-c9ed93c0beaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.168410 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.168387 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 15:35:48.168535 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.168414 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 15:35:48.168535 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.168439 2573 projected.go:196] Error preparing data for projected volume kube-api-access-z8fpb for pod openshift-network-diagnostics/network-check-target-8k54x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:48.168535 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.168531 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb podName:bea76fa1-573a-410a-958d-b63840be1aed nodeName:}" failed. No retries permitted until 2026-07-16 15:35:48.668514875 +0000 UTC m=+3.107041681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z8fpb" (UniqueName: "kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb") pod "network-check-target-8k54x" (UID: "bea76fa1-573a-410a-958d-b63840be1aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:48.172067 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.172007 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72q4h\" (UniqueName: \"kubernetes.io/projected/76bc2888-9278-47da-bec1-0efc06a0aaca-kube-api-access-72q4h\") pod \"iptables-alerter-mshv5\" (UID: \"76bc2888-9278-47da-bec1-0efc06a0aaca\") " pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.172229 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.172208 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfwf\" (UniqueName: \"kubernetes.io/projected/9295763d-a0e9-43f3-98c9-a86588ddf8f2-kube-api-access-vjfwf\") pod \"node-resolver-dh7cp\" (UID: \"9295763d-a0e9-43f3-98c9-a86588ddf8f2\") " pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.172594 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.172575 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpx84\" (UniqueName: \"kubernetes.io/projected/01037690-454d-4b4a-8b72-c9ed93c0beaf-kube-api-access-hpx84\") pod \"multus-additional-cni-plugins-md42f\" (UID: \"01037690-454d-4b4a-8b72-c9ed93c0beaf\") " pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.173148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.173129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblwx\" (UniqueName: \"kubernetes.io/projected/1619a010-0b97-427e-9d72-5d0b576077ba-kube-api-access-mblwx\") pod \"multus-nfwrq\" (UID: \"1619a010-0b97-427e-9d72-5d0b576077ba\") " pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.244362 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.244330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pllg4" Jul 16 15:35:48.252190 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.252166 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:35:48.252502 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:48.252477 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61456613_eddd_4594_b4fa_c61c23d2976d.slice/crio-439400880b2672b1bd64300e5a58d8d6f14417ad90fdfb26e6d06b4c5ac0c9f0 WatchSource:0}: Error finding container 439400880b2672b1bd64300e5a58d8d6f14417ad90fdfb26e6d06b4c5ac0c9f0: Status 404 returned error can't find the container with id 439400880b2672b1bd64300e5a58d8d6f14417ad90fdfb26e6d06b4c5ac0c9f0 Jul 16 15:35:48.260130 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:48.260110 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2308a825_c474_4f66_a4ca_12e024550bdf.slice/crio-dcd155283a478ad1e5b296758a64e68ac1bf19b056fa8538a24db5e25a72bac6 WatchSource:0}: Error finding container dcd155283a478ad1e5b296758a64e68ac1bf19b056fa8538a24db5e25a72bac6: Status 404 returned error can't find the container with id dcd155283a478ad1e5b296758a64e68ac1bf19b056fa8538a24db5e25a72bac6 Jul 16 15:35:48.260976 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.260939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:35:48.266713 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.266684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6s8bz" Jul 16 15:35:48.271212 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:48.271180 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065bbaff_e023_4be5_bd5b_be46329be9a3.slice/crio-7c008084a4edaaeb5bbc7b2af9ce4f49850f84b67102efbf528fca856c0d9226 WatchSource:0}: Error finding container 7c008084a4edaaeb5bbc7b2af9ce4f49850f84b67102efbf528fca856c0d9226: Status 404 returned error can't find the container with id 7c008084a4edaaeb5bbc7b2af9ce4f49850f84b67102efbf528fca856c0d9226 Jul 16 15:35:48.271962 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.271944 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 15:35:48.274624 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.274590 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" Jul 16 15:35:48.279082 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:48.279056 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf6f6e9_5c8d_4aa1_9d9a_dab9a39240d4.slice/crio-78a983968d22c2d76ac8afd7388ad38e52208a13a6e8b3ee5d92d3c8c54e5dc3 WatchSource:0}: Error finding container 78a983968d22c2d76ac8afd7388ad38e52208a13a6e8b3ee5d92d3c8c54e5dc3: Status 404 returned error can't find the container with id 78a983968d22c2d76ac8afd7388ad38e52208a13a6e8b3ee5d92d3c8c54e5dc3 Jul 16 15:35:48.283233 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.283213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-md42f" Jul 16 15:35:48.284035 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:48.284008 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94066e5_1ed5_4b3e_93d8_881ee4b3de4c.slice/crio-88ffa7e4866d1c419d03fb4fa4274dca3c9b08b1f2feec688a134c1c6967293e WatchSource:0}: Error finding container 88ffa7e4866d1c419d03fb4fa4274dca3c9b08b1f2feec688a134c1c6967293e: Status 404 returned error can't find the container with id 88ffa7e4866d1c419d03fb4fa4274dca3c9b08b1f2feec688a134c1c6967293e Jul 16 15:35:48.291046 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.291029 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dh7cp" Jul 16 15:35:48.300141 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.300000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mshv5" Jul 16 15:35:48.304362 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:48.304341 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9295763d_a0e9_43f3_98c9_a86588ddf8f2.slice/crio-e40f972bad211b350418fe0f28220bb4bc93dc863cf4982ecc62fa8ea49cd99b WatchSource:0}: Error finding container e40f972bad211b350418fe0f28220bb4bc93dc863cf4982ecc62fa8ea49cd99b: Status 404 returned error can't find the container with id e40f972bad211b350418fe0f28220bb4bc93dc863cf4982ecc62fa8ea49cd99b Jul 16 15:35:48.305961 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.305856 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nfwrq" Jul 16 15:35:48.310803 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:48.310774 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bc2888_9278_47da_bec1_0efc06a0aaca.slice/crio-e50853fc0b51c2834cfa2917137db689d5c9827ab076a04e0655e0c66cd60692 WatchSource:0}: Error finding container e50853fc0b51c2834cfa2917137db689d5c9827ab076a04e0655e0c66cd60692: Status 404 returned error can't find the container with id e50853fc0b51c2834cfa2917137db689d5c9827ab076a04e0655e0c66cd60692 Jul 16 15:35:48.318981 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:35:48.318958 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1619a010_0b97_427e_9d72_5d0b576077ba.slice/crio-aadaecbc32a2e74601e5139465e21e5bcdf54db5e20db6b086f9d3922d8ea3d3 WatchSource:0}: Error finding container aadaecbc32a2e74601e5139465e21e5bcdf54db5e20db6b086f9d3922d8ea3d3: Status 404 returned error can't find the container with id aadaecbc32a2e74601e5139465e21e5bcdf54db5e20db6b086f9d3922d8ea3d3 Jul 16 15:35:48.381921 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.381865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" event={"ID":"fc266b970294ca7e6b02e58f7dfbfaf7","Type":"ContainerStarted","Data":"34429859d677aacaab2f53e3d8fe52b7aa2a0052cad238edef49dd028a7ab339"} Jul 16 15:35:48.383180 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.383150 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nfwrq" event={"ID":"1619a010-0b97-427e-9d72-5d0b576077ba","Type":"ContainerStarted","Data":"aadaecbc32a2e74601e5139465e21e5bcdf54db5e20db6b086f9d3922d8ea3d3"} Jul 16 15:35:48.384573 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.384542 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" event={"ID":"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c","Type":"ContainerStarted","Data":"88ffa7e4866d1c419d03fb4fa4274dca3c9b08b1f2feec688a134c1c6967293e"} Jul 16 15:35:48.385821 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.385800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"dcd155283a478ad1e5b296758a64e68ac1bf19b056fa8538a24db5e25a72bac6"} Jul 16 15:35:48.387069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.387046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pllg4" event={"ID":"61456613-eddd-4594-b4fa-c61c23d2976d","Type":"ContainerStarted","Data":"439400880b2672b1bd64300e5a58d8d6f14417ad90fdfb26e6d06b4c5ac0c9f0"} Jul 16 15:35:48.388098 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.388076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" event={"ID":"b41c40da60a15333861b842213a29712","Type":"ContainerStarted","Data":"a521fd1256d191c968ebda3c65635cff61fab174adfd700703da9400b20edc6c"} Jul 16 15:35:48.389983 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.389963 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mshv5" event={"ID":"76bc2888-9278-47da-bec1-0efc06a0aaca","Type":"ContainerStarted","Data":"e50853fc0b51c2834cfa2917137db689d5c9827ab076a04e0655e0c66cd60692"} Jul 16 15:35:48.391601 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.391580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dh7cp" event={"ID":"9295763d-a0e9-43f3-98c9-a86588ddf8f2","Type":"ContainerStarted","Data":"e40f972bad211b350418fe0f28220bb4bc93dc863cf4982ecc62fa8ea49cd99b"} Jul 16 15:35:48.392737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.392716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md42f" event={"ID":"01037690-454d-4b4a-8b72-c9ed93c0beaf","Type":"ContainerStarted","Data":"ccf49cb401deec3818260bc424123305e0201e76305a30d9e9bd1d97304c9f9d"} Jul 16 15:35:48.393785 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.393741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6s8bz" event={"ID":"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4","Type":"ContainerStarted","Data":"78a983968d22c2d76ac8afd7388ad38e52208a13a6e8b3ee5d92d3c8c54e5dc3"} Jul 16 15:35:48.395036 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.394993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qxxr2" event={"ID":"065bbaff-e023-4be5-bd5b-be46329be9a3","Type":"ContainerStarted","Data":"7c008084a4edaaeb5bbc7b2af9ce4f49850f84b67102efbf528fca856c0d9226"} Jul 16 15:35:48.562215 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.562119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:48.562376 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.562260 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:48.562376 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.562330 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs podName:b8921e8d-2506-4ff1-abc3-21abb1038d4c nodeName:}" failed. No retries permitted until 2026-07-16 15:35:49.562311502 +0000 UTC m=+4.000838304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs") pod "network-metrics-daemon-nz489" (UID: "b8921e8d-2506-4ff1-abc3-21abb1038d4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:48.663045 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.663010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:48.663201 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.663187 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:48.663278 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.663249 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret podName:e7962491-319d-4d1c-8d70-74706e135117 nodeName:}" failed. No retries permitted until 2026-07-16 15:35:49.663229206 +0000 UTC m=+4.101756012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret") pod "global-pull-secret-syncer-sdfct" (UID: "e7962491-319d-4d1c-8d70-74706e135117") : object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:48.764181 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.764138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:48.764348 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.764291 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 15:35:48.764348 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.764311 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 15:35:48.764348 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.764324 2573 projected.go:196] Error preparing data for projected volume kube-api-access-z8fpb for pod openshift-network-diagnostics/network-check-target-8k54x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:48.764554 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:48.764387 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb podName:bea76fa1-573a-410a-958d-b63840be1aed nodeName:}" failed. No retries permitted until 2026-07-16 15:35:49.76436797 +0000 UTC m=+4.202894788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8fpb" (UniqueName: "kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb") pod "network-check-target-8k54x" (UID: "bea76fa1-573a-410a-958d-b63840be1aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:48.985602 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.985512 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 15:30:47 +0000 UTC" deadline="2028-01-24 07:19:35.712923064 +0000 UTC" Jul 16 15:35:48.985602 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:48.985561 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13359h43m46.727379498s" Jul 16 15:35:49.378065 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:49.377989 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:49.378229 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.378113 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:35:49.378679 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:49.378516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:49.378679 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.378629 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:35:49.571953 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:49.571283 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:49.571953 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.571442 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:49.571953 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.571512 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs podName:b8921e8d-2506-4ff1-abc3-21abb1038d4c nodeName:}" failed. No retries permitted until 2026-07-16 15:35:51.571492624 +0000 UTC m=+6.010019427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs") pod "network-metrics-daemon-nz489" (UID: "b8921e8d-2506-4ff1-abc3-21abb1038d4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:49.672874 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:49.672784 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:49.673081 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.673037 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:49.673147 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.673106 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret podName:e7962491-319d-4d1c-8d70-74706e135117 nodeName:}" failed. No retries permitted until 2026-07-16 15:35:51.673088211 +0000 UTC m=+6.111615033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret") pod "global-pull-secret-syncer-sdfct" (UID: "e7962491-319d-4d1c-8d70-74706e135117") : object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:49.774334 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:49.774044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:49.774334 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.774223 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 15:35:49.774334 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.774242 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 15:35:49.774334 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.774254 2573 projected.go:196] Error preparing data for projected volume kube-api-access-z8fpb for pod openshift-network-diagnostics/network-check-target-8k54x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:49.774334 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:49.774308 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb podName:bea76fa1-573a-410a-958d-b63840be1aed nodeName:}" failed. No retries permitted until 2026-07-16 15:35:51.774290243 +0000 UTC m=+6.212817047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8fpb" (UniqueName: "kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb") pod "network-check-target-8k54x" (UID: "bea76fa1-573a-410a-958d-b63840be1aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:49.986281 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:49.985937 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 15:30:47 +0000 UTC" deadline="2028-01-20 11:40:42.116561157 +0000 UTC" Jul 16 15:35:49.986281 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:49.986197 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13268h4m52.130369883s" Jul 16 15:35:50.378642 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:50.378593 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:50.378759 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:50.378738 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:35:50.417808 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:50.417115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" event={"ID":"fc266b970294ca7e6b02e58f7dfbfaf7","Type":"ContainerStarted","Data":"25dbed453de3250a7780aaddbeaf51cb2ed8b9e90bb24fe8a14873719cc53249"} Jul 16 15:35:51.378381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:51.377765 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:51.378381 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.377906 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:35:51.378381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:51.377973 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:51.378381 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.378048 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:35:51.428246 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:51.428211 2573 generic.go:356] "Generic (PLEG): container finished" podID="b41c40da60a15333861b842213a29712" containerID="d26bc50d7be1531a749a46327d95170acbf9f5321f3b7e611cc220b7c312da03" exitCode=0 Jul 16 15:35:51.428422 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:51.428298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" event={"ID":"b41c40da60a15333861b842213a29712","Type":"ContainerDied","Data":"d26bc50d7be1531a749a46327d95170acbf9f5321f3b7e611cc220b7c312da03"} Jul 16 15:35:51.451988 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:51.451944 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-48.ec2.internal" podStartSLOduration=4.451928635 podStartE2EDuration="4.451928635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:35:50.436739033 +0000 UTC m=+4.875265857" watchObservedRunningTime="2026-07-16 15:35:51.451928635 +0000 UTC m=+5.890455457" Jul 16 15:35:51.586248 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:51.586207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:51.586430 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.586376 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:51.586497 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.586444 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs podName:b8921e8d-2506-4ff1-abc3-21abb1038d4c nodeName:}" failed. No retries permitted until 2026-07-16 15:35:55.586421569 +0000 UTC m=+10.024948384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs") pod "network-metrics-daemon-nz489" (UID: "b8921e8d-2506-4ff1-abc3-21abb1038d4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:51.686841 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:51.686762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:51.687083 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.687064 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:51.687165 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.687151 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret podName:e7962491-319d-4d1c-8d70-74706e135117 nodeName:}" failed. No retries permitted until 2026-07-16 15:35:55.687121833 +0000 UTC m=+10.125648637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret") pod "global-pull-secret-syncer-sdfct" (UID: "e7962491-319d-4d1c-8d70-74706e135117") : object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:51.788067 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:51.788032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:51.788255 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.788236 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 15:35:51.788311 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.788265 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 15:35:51.788311 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.788278 2573 projected.go:196] Error preparing data for projected volume kube-api-access-z8fpb for pod openshift-network-diagnostics/network-check-target-8k54x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:51.788421 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:51.788333 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb podName:bea76fa1-573a-410a-958d-b63840be1aed nodeName:}" failed. No retries permitted until 2026-07-16 15:35:55.78831421 +0000 UTC m=+10.226841014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8fpb" (UniqueName: "kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb") pod "network-check-target-8k54x" (UID: "bea76fa1-573a-410a-958d-b63840be1aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:52.378019 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:52.377986 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:52.378203 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:52.378137 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:35:53.378116 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:53.378087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:53.378597 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:53.378189 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:35:53.378597 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:53.378246 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:53.378597 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:53.378357 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:35:54.377754 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:54.377720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:54.377922 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:54.377859 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:35:55.378200 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:55.378099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:55.378651 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.378258 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:35:55.378651 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:55.378385 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:55.378651 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.378495 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:35:55.620856 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:55.620818 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:55.621037 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.620961 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:55.621037 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.621022 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs podName:b8921e8d-2506-4ff1-abc3-21abb1038d4c nodeName:}" failed. No retries permitted until 2026-07-16 15:36:03.621003565 +0000 UTC m=+18.059530380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs") pod "network-metrics-daemon-nz489" (UID: "b8921e8d-2506-4ff1-abc3-21abb1038d4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:35:55.721878 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:55.721789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:55.722038 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.721971 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:55.722038 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.722030 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret podName:e7962491-319d-4d1c-8d70-74706e135117 nodeName:}" failed. No retries permitted until 2026-07-16 15:36:03.722011959 +0000 UTC m=+18.160538776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret") pod "global-pull-secret-syncer-sdfct" (UID: "e7962491-319d-4d1c-8d70-74706e135117") : object "kube-system"/"original-pull-secret" not registered Jul 16 15:35:55.823361 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:55.822969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:55.823361 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.823178 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 15:35:55.823361 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.823200 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 15:35:55.823361 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.823211 2573 projected.go:196] Error preparing data for projected volume kube-api-access-z8fpb for pod openshift-network-diagnostics/network-check-target-8k54x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:55.823361 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:55.823263 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb podName:bea76fa1-573a-410a-958d-b63840be1aed nodeName:}" failed. No retries permitted until 2026-07-16 15:36:03.823247318 +0000 UTC m=+18.261774119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8fpb" (UniqueName: "kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb") pod "network-check-target-8k54x" (UID: "bea76fa1-573a-410a-958d-b63840be1aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:35:56.379067 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:56.379033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:56.379480 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:56.379160 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:35:57.378287 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:57.378194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:57.378444 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:57.378194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:57.378444 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:57.378315 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:35:57.378444 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:57.378370 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:35:58.378168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:58.378141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:35:58.378508 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:58.378252 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:35:59.377428 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:59.377396 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:35:59.377610 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:35:59.377396 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:35:59.377610 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:59.377493 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:35:59.377610 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:35:59.377580 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:00.377849 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:00.377801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:00.378415 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:00.377985 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:01.377578 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:01.377541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:01.377786 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:01.377682 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:01.377786 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:01.377551 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:01.377888 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:01.377823 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:02.378071 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:02.378038 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:02.378518 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:02.378168 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:03.377893 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:03.377860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:03.378091 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.378010 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:03.378091 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:03.378023 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:03.378389 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.378140 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:03.680965 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:03.680930 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:03.681125 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.681059 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:36:03.681167 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.681133 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs podName:b8921e8d-2506-4ff1-abc3-21abb1038d4c nodeName:}" failed. No retries permitted until 2026-07-16 15:36:19.681113365 +0000 UTC m=+34.119640168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs") pod "network-metrics-daemon-nz489" (UID: "b8921e8d-2506-4ff1-abc3-21abb1038d4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:36:03.781599 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:03.781562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:03.781788 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.781728 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 15:36:03.781848 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.781806 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret podName:e7962491-319d-4d1c-8d70-74706e135117 nodeName:}" failed. No retries permitted until 2026-07-16 15:36:19.781783096 +0000 UTC m=+34.220309901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret") pod "global-pull-secret-syncer-sdfct" (UID: "e7962491-319d-4d1c-8d70-74706e135117") : object "kube-system"/"original-pull-secret" not registered Jul 16 15:36:03.882666 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:03.882632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:03.882879 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.882856 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 15:36:03.882947 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.882886 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 15:36:03.882947 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.882903 2573 projected.go:196] Error preparing data for projected volume kube-api-access-z8fpb for pod openshift-network-diagnostics/network-check-target-8k54x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:36:03.883036 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:03.882978 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb podName:bea76fa1-573a-410a-958d-b63840be1aed nodeName:}" failed. No retries permitted until 2026-07-16 15:36:19.882957537 +0000 UTC m=+34.321484344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8fpb" (UniqueName: "kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb") pod "network-check-target-8k54x" (UID: "bea76fa1-573a-410a-958d-b63840be1aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:36:04.378231 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:04.378197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:04.378670 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:04.378328 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:05.378036 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:05.378005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:05.378215 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:05.378045 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:05.378215 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:05.378128 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:05.378634 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:05.378245 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:06.378507 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:06.378476 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:06.378935 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:06.378595 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:07.377379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:07.377341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:07.377563 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:07.377341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:07.377563 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:07.377486 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:07.377696 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:07.377607 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:08.378487 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.378297 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:08.378978 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:08.378566 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:08.459091 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.459063 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dh7cp" event={"ID":"9295763d-a0e9-43f3-98c9-a86588ddf8f2","Type":"ContainerStarted","Data":"08e73d158e6a9e5f2ce5c2a660a21b6edb051d2c0d32ff00fb0c0f47b6de2875"} Jul 16 15:36:08.460321 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.460300 2573 generic.go:356] "Generic (PLEG): container finished" podID="01037690-454d-4b4a-8b72-c9ed93c0beaf" containerID="075cf8810d309c061b694555a1c82c11fa0aef6c45eba64c25a80acfb7bfd120" exitCode=0 Jul 16 15:36:08.460415 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.460364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md42f" event={"ID":"01037690-454d-4b4a-8b72-c9ed93c0beaf","Type":"ContainerDied","Data":"075cf8810d309c061b694555a1c82c11fa0aef6c45eba64c25a80acfb7bfd120"} Jul 16 15:36:08.461919 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.461892 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6s8bz" event={"ID":"cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4","Type":"ContainerStarted","Data":"b4475519cbc3af80396da15a83a1d9a3f5570e2e70569452ed8809bc8ad684d9"} Jul 16 15:36:08.463285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.463264 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qxxr2" event={"ID":"065bbaff-e023-4be5-bd5b-be46329be9a3","Type":"ContainerStarted","Data":"f61e9993de60ca49ede10f642a94957a177ce5562341da41fd0a1693410df610"} Jul 16 15:36:08.464850 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.464831 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nfwrq" event={"ID":"1619a010-0b97-427e-9d72-5d0b576077ba","Type":"ContainerStarted","Data":"550aaee4a7925391f554f03ff3e4ad34f7903cdae5c99eded11e3e0e28e48e2b"} Jul 16 15:36:08.466104 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.466083 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" event={"ID":"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c","Type":"ContainerStarted","Data":"a650a791df84b82f0882905b482e7cd2cdc0ff74a17efd62055642fcd731c0e4"} Jul 16 15:36:08.467434 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.467420 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:36:08.467754 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.467737 2573 generic.go:356] "Generic (PLEG): container finished" podID="2308a825-c474-4f66-a4ca-12e024550bdf" containerID="8580fd111da696ada7378935b5283b54832fd0c5704f769aee79443fbe1a4b4d" exitCode=1 Jul 16 15:36:08.467828 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.467796 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerDied","Data":"8580fd111da696ada7378935b5283b54832fd0c5704f769aee79443fbe1a4b4d"} Jul 16 15:36:08.467866 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.467848 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"863f9cd0786a9fc2d2862ca5eb736adbebbcf7c0d779bbe55cc741cfeba37d3f"} Jul 16 15:36:08.468976 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.468931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pllg4" event={"ID":"61456613-eddd-4594-b4fa-c61c23d2976d","Type":"ContainerStarted","Data":"5b2c4558d44c64d7761adedd89aab8b2a17c9177102903126be62d56a83b3e03"} Jul 16 15:36:08.470366 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.470348 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" event={"ID":"b41c40da60a15333861b842213a29712","Type":"ContainerStarted","Data":"f924f310dc2d6176e3fc85ee0efbfadf1365beb08fff758e62e7e05e07680a3b"} Jul 16 15:36:08.477728 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.477691 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dh7cp" podStartSLOduration=2.985558687 podStartE2EDuration="22.477679587s" totalImagesPullingTime="19.4921209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:08.477203205 +0000 UTC m=+22.915730029" watchObservedRunningTime="2026-07-16 15:36:08.477679587 +0000 UTC m=+22.916206408" Jul 16 15:36:08.524809 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.524767 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nfwrq" podStartSLOduration=2.995302134 podStartE2EDuration="22.524752716s" totalImagesPullingTime="19.529450582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:08.501373193 +0000 UTC m=+22.939900015" watchObservedRunningTime="2026-07-16 15:36:08.524752716 +0000 UTC m=+22.963279516" Jul 16 15:36:08.525357 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.525332 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pllg4" podStartSLOduration=2.950720235 podStartE2EDuration="22.525322316s" totalImagesPullingTime="19.574602081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:08.524840141 +0000 UTC m=+22.963366963" watchObservedRunningTime="2026-07-16 15:36:08.525322316 +0000 UTC m=+22.963849136" Jul 16 15:36:08.541262 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.541225 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6s8bz" podStartSLOduration=7.662458787 podStartE2EDuration="22.541212331s" totalImagesPullingTime="14.878753544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:08.541140447 +0000 UTC m=+22.979667274" watchObservedRunningTime="2026-07-16 15:36:08.541212331 +0000 UTC m=+22.979739148" Jul 16 15:36:08.608235 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.608102 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-48.ec2.internal" podStartSLOduration=21.608088655 podStartE2EDuration="21.608088655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:08.588421634 +0000 UTC m=+23.026948459" watchObservedRunningTime="2026-07-16 15:36:08.608088655 +0000 UTC m=+23.046615477" Jul 16 15:36:08.608667 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.608645 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qxxr2" podStartSLOduration=3.059102803 podStartE2EDuration="22.608636775s" totalImagesPullingTime="19.549533972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:08.607666431 +0000 UTC m=+23.046193254" watchObservedRunningTime="2026-07-16 15:36:08.608636775 +0000 UTC m=+23.047163595" Jul 16 15:36:08.969473 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:08.969448 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6s8bz_cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4/node-ca/0.log" Jul 16 15:36:09.377979 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.377950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:09.378152 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.377950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:09.378152 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:09.378064 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:09.378257 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:09.378170 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:09.475898 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.475867 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:36:09.476446 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.476345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"664f6a369c73c238dd62a91ef47c8693e0ba8e58d733a1f68de5b37cdecffc70"} Jul 16 15:36:09.476446 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.476428 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"05a7e2d8e9e999c4a30459e2c6af38e493af0ac0766a7de0b486fd4e62dde35e"} Jul 16 15:36:09.476546 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.476465 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"8aa76cd9a3c6ff74c333ff460dd2cc0e725e8b679f09a081767980d5a7b16535"} Jul 16 15:36:09.476546 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.476493 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"bbf9fb734f769e777fea072650407d7d6de5b1d2ba3b6cb76d14890eab78d0dc"} Jul 16 15:36:09.477951 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.477909 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mshv5" event={"ID":"76bc2888-9278-47da-bec1-0efc06a0aaca","Type":"ContainerStarted","Data":"01a8835eb28e52bd839f647ae642457f6fec63bef0be7efde451cf720ffd9a78"} Jul 16 15:36:09.497168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:09.497114 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mshv5" podStartSLOduration=3.9883091090000002 podStartE2EDuration="23.497096286s" totalImagesPullingTime="19.508787177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:09.497048463 +0000 UTC m=+23.935575286" watchObservedRunningTime="2026-07-16 15:36:09.497096286 +0000 UTC m=+23.935623112" Jul 16 15:36:10.377404 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:10.377378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:10.377555 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:10.377515 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:10.500032 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:10.500002 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 15:36:11.244249 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.244113 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T15:36:10.500024667Z","UUID":"63e55f9c-5358-40f6-9d0b-a7e741bb94e9","Handler":null,"Name":"","Endpoint":""} Jul 16 15:36:11.246113 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.245940 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 15:36:11.246249 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.246127 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 15:36:11.377805 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.377776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:11.377978 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.377789 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:11.377978 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:11.377901 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:11.377978 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:11.377949 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:11.486549 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.486519 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:36:11.487002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.486968 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"9d888500fcc60165b452689a8debc5bd6f61b2dd3c1368c08a45e652287af8e7"} Jul 16 15:36:11.488802 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.488780 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" event={"ID":"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c","Type":"ContainerStarted","Data":"43d9d21a2f15870612ea4fa7860daa8c5ddf70de34740a0509e5beb974239e2f"} Jul 16 15:36:11.630443 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.630411 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:36:11.631083 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:11.631062 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:36:12.378159 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:12.378130 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:12.378324 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:12.378254 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:13.377320 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.377208 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:13.377856 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.377213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:13.377856 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:13.377385 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:13.377856 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:13.377460 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:13.499513 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.499480 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:36:13.499977 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.499945 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"6275afb4c99f8db15f7041364cd2af63835bc64db42572ce4e6c0626531b141f"} Jul 16 15:36:13.500322 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.500247 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:36:13.500322 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.500319 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:36:13.500603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.500465 2573 scope.go:117] "RemoveContainer" containerID="8580fd111da696ada7378935b5283b54832fd0c5704f769aee79443fbe1a4b4d" Jul 16 15:36:13.502575 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.502465 2573 generic.go:356] "Generic (PLEG): container finished" podID="01037690-454d-4b4a-8b72-c9ed93c0beaf" containerID="c05b1f35f49e05169367184aff711b6ae0a0ea93a7f9071c2adbf4b1c5c1bb86" exitCode=0 Jul 16 15:36:13.502575 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.502506 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md42f" event={"ID":"01037690-454d-4b4a-8b72-c9ed93c0beaf","Type":"ContainerDied","Data":"c05b1f35f49e05169367184aff711b6ae0a0ea93a7f9071c2adbf4b1c5c1bb86"} Jul 16 15:36:13.520773 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.520745 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:36:13.521029 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:13.521009 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:36:14.377668 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:14.377627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:14.378229 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:14.377762 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:14.506846 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:14.506809 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" event={"ID":"d94066e5-1ed5-4b3e-93d8-881ee4b3de4c","Type":"ContainerStarted","Data":"b0238f0d5196acc44dec34c6fb0be6e0fd14602cd74ca83748f15418bda01a76"} Jul 16 15:36:14.509979 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:14.509961 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:36:14.510305 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:14.510281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" event={"ID":"2308a825-c474-4f66-a4ca-12e024550bdf","Type":"ContainerStarted","Data":"bd06bc2e627ec8d4a7e999a684425b4a955f1492fb2b3abe074471a387a80697"} Jul 16 15:36:14.510498 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:14.510483 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:36:14.525587 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:14.525548 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r8ttw" podStartSLOduration=3.766864815 podStartE2EDuration="28.525535435s" totalImagesPullingTime="24.75867062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:14.524221376 +0000 UTC m=+28.962748221" watchObservedRunningTime="2026-07-16 15:36:14.525535435 +0000 UTC m=+28.964062258" Jul 16 15:36:14.551850 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:14.551806 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" podStartSLOduration=8.639584303 podStartE2EDuration="28.55178878s" totalImagesPullingTime="19.912204477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:14.550393073 +0000 UTC m=+28.988919896" watchObservedRunningTime="2026-07-16 15:36:14.55178878 +0000 UTC m=+28.990315606" Jul 16 15:36:15.378171 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:15.378138 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:15.378642 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:15.378241 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:15.378642 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:15.378308 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:15.378642 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:15.378396 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:15.514319 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:15.514289 2573 generic.go:356] "Generic (PLEG): container finished" podID="01037690-454d-4b4a-8b72-c9ed93c0beaf" containerID="f5e2bd409ddae5a2c27d542854db748fc708853eebb5b6253b825b59e4665004" exitCode=0 Jul 16 15:36:15.514476 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:15.514377 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md42f" event={"ID":"01037690-454d-4b4a-8b72-c9ed93c0beaf","Type":"ContainerDied","Data":"f5e2bd409ddae5a2c27d542854db748fc708853eebb5b6253b825b59e4665004"} Jul 16 15:36:16.379122 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:16.379089 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:16.379466 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:16.379210 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:17.377586 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:17.377556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:17.377914 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:17.377671 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:17.377914 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:17.377729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:17.377914 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:17.377817 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:17.455410 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:17.455384 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:36:17.455791 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:17.455487 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 16 15:36:17.455939 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:17.455920 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qxxr2" Jul 16 15:36:17.520665 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:17.520636 2573 generic.go:356] "Generic (PLEG): container finished" podID="01037690-454d-4b4a-8b72-c9ed93c0beaf" containerID="1c4924043bfc4054fcd2001a0d47f888cf5a49703137cbc7b6dc30e8bfbeb9a4" exitCode=0 Jul 16 15:36:17.520843 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:17.520711 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md42f" event={"ID":"01037690-454d-4b4a-8b72-c9ed93c0beaf","Type":"ContainerDied","Data":"1c4924043bfc4054fcd2001a0d47f888cf5a49703137cbc7b6dc30e8bfbeb9a4"} Jul 16 15:36:18.377508 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:18.377474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:18.377705 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:18.377638 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:19.378051 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:19.377861 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:19.378508 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:19.377896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:19.378508 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.378140 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:19.378508 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.378250 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:19.695815 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:19.695727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:19.695979 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.695901 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:36:19.695979 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.695974 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs podName:b8921e8d-2506-4ff1-abc3-21abb1038d4c nodeName:}" failed. No retries permitted until 2026-07-16 15:36:51.695953614 +0000 UTC m=+66.134480418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs") pod "network-metrics-daemon-nz489" (UID: "b8921e8d-2506-4ff1-abc3-21abb1038d4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 15:36:19.796684 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:19.796657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:19.796844 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.796806 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 15:36:19.796902 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.796872 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret podName:e7962491-319d-4d1c-8d70-74706e135117 nodeName:}" failed. No retries permitted until 2026-07-16 15:36:51.796854678 +0000 UTC m=+66.235381493 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret") pod "global-pull-secret-syncer-sdfct" (UID: "e7962491-319d-4d1c-8d70-74706e135117") : object "kube-system"/"original-pull-secret" not registered Jul 16 15:36:19.897084 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:19.897053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:19.897247 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.897222 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 15:36:19.897247 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.897242 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 15:36:19.897346 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.897252 2573 projected.go:196] Error preparing data for projected volume kube-api-access-z8fpb for pod openshift-network-diagnostics/network-check-target-8k54x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:36:19.897346 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:19.897306 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb podName:bea76fa1-573a-410a-958d-b63840be1aed nodeName:}" failed. No retries permitted until 2026-07-16 15:36:51.897289036 +0000 UTC m=+66.335815837 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8fpb" (UniqueName: "kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb") pod "network-check-target-8k54x" (UID: "bea76fa1-573a-410a-958d-b63840be1aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 15:36:20.379453 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:20.378390 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:20.379453 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:20.379206 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:21.378019 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:21.377991 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:21.378201 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:21.378103 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:21.378201 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:21.378166 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:21.378315 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:21.378288 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:22.377737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:22.377698 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:22.378109 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:22.377838 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:22.534534 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:22.534504 2573 generic.go:356] "Generic (PLEG): container finished" podID="01037690-454d-4b4a-8b72-c9ed93c0beaf" containerID="b7de640753c3d0371b364272a53f6077be5d39f1a45fd9b918a0b54f792e00fc" exitCode=0 Jul 16 15:36:22.534693 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:22.534543 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md42f" event={"ID":"01037690-454d-4b4a-8b72-c9ed93c0beaf","Type":"ContainerDied","Data":"b7de640753c3d0371b364272a53f6077be5d39f1a45fd9b918a0b54f792e00fc"} Jul 16 15:36:23.377516 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:23.377490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:23.377688 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:23.377490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:23.377688 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:23.377592 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:23.377688 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:23.377668 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:23.542769 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:23.542737 2573 generic.go:356] "Generic (PLEG): container finished" podID="01037690-454d-4b4a-8b72-c9ed93c0beaf" containerID="2a26af43bc3665cf26a10844673f9ecff3319dc11530150eff9f296365aae0c7" exitCode=0 Jul 16 15:36:23.542915 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:23.542780 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md42f" event={"ID":"01037690-454d-4b4a-8b72-c9ed93c0beaf","Type":"ContainerDied","Data":"2a26af43bc3665cf26a10844673f9ecff3319dc11530150eff9f296365aae0c7"} Jul 16 15:36:24.377501 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:24.377478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:24.377690 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:24.377578 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:24.547410 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:24.547377 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md42f" event={"ID":"01037690-454d-4b4a-8b72-c9ed93c0beaf","Type":"ContainerStarted","Data":"d1b3d74783fdd49a4d7e84bc00d730aa2fc04f3c48451c5613d6ae73cad0cb47"} Jul 16 15:36:24.570190 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:24.570142 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-md42f" podStartSLOduration=8.107705783 podStartE2EDuration="38.570128889s" totalImagesPullingTime="30.462423106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:24.568485777 +0000 UTC m=+39.007012599" watchObservedRunningTime="2026-07-16 15:36:24.570128889 +0000 UTC m=+39.008655713" Jul 16 15:36:25.377511 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:25.377484 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:25.377696 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:25.377485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:25.377696 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:25.377579 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:25.377696 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:25.377679 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:26.378451 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:26.378421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:26.378825 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:26.378505 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:27.377776 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:27.377747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:27.377776 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:27.377781 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:27.377972 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:27.377848 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:27.378010 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:27.377961 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:28.377732 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:28.377703 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:28.378146 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:28.377810 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:29.378248 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:29.378221 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:29.378692 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:29.378258 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:29.378692 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:29.378353 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:29.378692 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:29.378456 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:30.378231 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:30.378200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:30.378386 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:30.378343 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:31.377438 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:31.377406 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:31.377678 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:31.377451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:31.377678 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:31.377514 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:31.377678 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:31.377635 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:32.378053 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:32.378017 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:32.378527 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:32.378129 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:33.377772 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:33.377749 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:33.377901 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:33.377787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:33.377901 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:33.377885 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:33.378054 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:33.378023 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:33.487009 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:33.486978 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nz489"] Jul 16 15:36:33.487824 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:33.487101 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:33.487824 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:33.487226 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:33.488863 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:33.488747 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8k54x"] Jul 16 15:36:33.493369 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:33.493348 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sdfct"] Jul 16 15:36:33.565117 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:33.565090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:33.565264 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:33.565090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:33.565264 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:33.565199 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:33.565349 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:33.565256 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:35.377975 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:35.377807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:35.378270 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:35.377863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:35.378270 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:35.378051 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:35.378270 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:35.378143 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:35.378270 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:35.377869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:35.378270 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:35.378230 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:37.377878 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.377855 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:37.378148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.377875 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:37.378148 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:37.377956 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nz489" podUID="b8921e8d-2506-4ff1-abc3-21abb1038d4c" Jul 16 15:36:37.378148 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:37.378041 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdfct" podUID="e7962491-319d-4d1c-8d70-74706e135117" Jul 16 15:36:37.378148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.378090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:37.378148 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:37.378137 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k54x" podUID="bea76fa1-573a-410a-958d-b63840be1aed" Jul 16 15:36:37.817708 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.817682 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-48.ec2.internal" event="NodeReady" Jul 16 15:36:37.817862 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.817798 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 15:36:37.893375 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.893344 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tr9ts"] Jul 16 15:36:37.914711 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.914684 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kfqqs"] Jul 16 15:36:37.914845 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.914805 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:37.918891 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.918872 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 15:36:37.919076 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.919065 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vskms\"" Jul 16 15:36:37.919393 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.919377 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 15:36:37.936834 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.936814 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tr9ts"] Jul 16 15:36:37.936923 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.936843 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kfqqs"] Jul 16 15:36:37.936961 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.936926 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:37.941298 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.941231 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 15:36:37.941415 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.941325 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 15:36:37.941673 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.941658 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 15:36:37.945707 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.943106 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zzsd4\"" Jul 16 15:36:37.950157 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:37.950143 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 15:36:38.001129 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.001101 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t99dx"] Jul 16 15:36:38.022941 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.022920 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t99dx"] Jul 16 15:36:38.023017 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.022998 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t99dx" Jul 16 15:36:38.026479 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.026460 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 15:36:38.026579 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.026482 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 15:36:38.026685 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.026671 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 15:36:38.026806 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.026791 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6gbrz\"" Jul 16 15:36:38.030432 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.030413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e5984f0-0dcb-467c-876d-84c7943edc28-metrics-tls\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.030516 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.030456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvd2p\" (UniqueName: \"kubernetes.io/projected/0e5984f0-0dcb-467c-876d-84c7943edc28-kube-api-access-bvd2p\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.030560 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.030509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5984f0-0dcb-467c-876d-84c7943edc28-config-volume\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.030560 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.030539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5984f0-0dcb-467c-876d-84c7943edc28-tmp-dir\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.131765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a58c255-8ae3-46f3-977e-310e0db54caa-cert\") pod \"ingress-canary-t99dx\" (UID: \"1a58c255-8ae3-46f3-977e-310e0db54caa\") " pod="openshift-ingress-canary/ingress-canary-t99dx" Jul 16 15:36:38.131765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5065548b-266f-4194-b8fe-f44fb431cdd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.131765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5065548b-266f-4194-b8fe-f44fb431cdd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.132002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131774 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5065548b-266f-4194-b8fe-f44fb431cdd4-crio-socket\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.132002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131789 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5065548b-266f-4194-b8fe-f44fb431cdd4-data-volume\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.132002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kj5z\" (UniqueName: \"kubernetes.io/projected/1a58c255-8ae3-46f3-977e-310e0db54caa-kube-api-access-4kj5z\") pod \"ingress-canary-t99dx\" (UID: \"1a58c255-8ae3-46f3-977e-310e0db54caa\") " pod="openshift-ingress-canary/ingress-canary-t99dx" Jul 16 15:36:38.132002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e5984f0-0dcb-467c-876d-84c7943edc28-metrics-tls\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.132002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvd2p\" (UniqueName: \"kubernetes.io/projected/0e5984f0-0dcb-467c-876d-84c7943edc28-kube-api-access-bvd2p\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.132002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsnp\" (UniqueName: \"kubernetes.io/projected/5065548b-266f-4194-b8fe-f44fb431cdd4-kube-api-access-8hsnp\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.132002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5984f0-0dcb-467c-876d-84c7943edc28-config-volume\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.132002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.131972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5984f0-0dcb-467c-876d-84c7943edc28-tmp-dir\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.132307 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.132255 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5984f0-0dcb-467c-876d-84c7943edc28-tmp-dir\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.132439 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.132411 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5984f0-0dcb-467c-876d-84c7943edc28-config-volume\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.135838 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.135820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e5984f0-0dcb-467c-876d-84c7943edc28-metrics-tls\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.141677 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.141659 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvd2p\" (UniqueName: \"kubernetes.io/projected/0e5984f0-0dcb-467c-876d-84c7943edc28-kube-api-access-bvd2p\") pod \"dns-default-tr9ts\" (UID: \"0e5984f0-0dcb-467c-876d-84c7943edc28\") " pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.224156 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.224115 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:38.232367 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsnp\" (UniqueName: \"kubernetes.io/projected/5065548b-266f-4194-b8fe-f44fb431cdd4-kube-api-access-8hsnp\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.232432 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a58c255-8ae3-46f3-977e-310e0db54caa-cert\") pod \"ingress-canary-t99dx\" (UID: \"1a58c255-8ae3-46f3-977e-310e0db54caa\") " pod="openshift-ingress-canary/ingress-canary-t99dx" Jul 16 15:36:38.232432 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5065548b-266f-4194-b8fe-f44fb431cdd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.232533 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5065548b-266f-4194-b8fe-f44fb431cdd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.232533 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5065548b-266f-4194-b8fe-f44fb431cdd4-crio-socket\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.232533 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232476 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5065548b-266f-4194-b8fe-f44fb431cdd4-data-volume\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.232533 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kj5z\" (UniqueName: \"kubernetes.io/projected/1a58c255-8ae3-46f3-977e-310e0db54caa-kube-api-access-4kj5z\") pod \"ingress-canary-t99dx\" (UID: \"1a58c255-8ae3-46f3-977e-310e0db54caa\") " pod="openshift-ingress-canary/ingress-canary-t99dx" Jul 16 15:36:38.232754 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232649 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5065548b-266f-4194-b8fe-f44fb431cdd4-crio-socket\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.232878 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5065548b-266f-4194-b8fe-f44fb431cdd4-data-volume\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.233014 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.232995 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5065548b-266f-4194-b8fe-f44fb431cdd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.234713 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.234693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5065548b-266f-4194-b8fe-f44fb431cdd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.234811 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.234797 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a58c255-8ae3-46f3-977e-310e0db54caa-cert\") pod \"ingress-canary-t99dx\" (UID: \"1a58c255-8ae3-46f3-977e-310e0db54caa\") " pod="openshift-ingress-canary/ingress-canary-t99dx" Jul 16 15:36:38.244128 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.244097 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kj5z\" (UniqueName: \"kubernetes.io/projected/1a58c255-8ae3-46f3-977e-310e0db54caa-kube-api-access-4kj5z\") pod \"ingress-canary-t99dx\" (UID: \"1a58c255-8ae3-46f3-977e-310e0db54caa\") " pod="openshift-ingress-canary/ingress-canary-t99dx" Jul 16 15:36:38.245897 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.245875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsnp\" (UniqueName: \"kubernetes.io/projected/5065548b-266f-4194-b8fe-f44fb431cdd4-kube-api-access-8hsnp\") pod \"insights-runtime-extractor-kfqqs\" (UID: \"5065548b-266f-4194-b8fe-f44fb431cdd4\") " pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.247787 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.247757 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kfqqs" Jul 16 15:36:38.330809 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.330782 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t99dx" Jul 16 15:36:38.434207 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.434180 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kfqqs"] Jul 16 15:36:38.440110 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.439883 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tr9ts"] Jul 16 15:36:38.444601 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:38.444574 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5984f0_0dcb_467c_876d_84c7943edc28.slice/crio-c596302d26cf01f3e54500666bb00bb00eb2cdb94310c2286fe1a8fc334c1aef WatchSource:0}: Error finding container c596302d26cf01f3e54500666bb00bb00eb2cdb94310c2286fe1a8fc334c1aef: Status 404 returned error can't find the container with id c596302d26cf01f3e54500666bb00bb00eb2cdb94310c2286fe1a8fc334c1aef Jul 16 15:36:38.487603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.487576 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t99dx"] Jul 16 15:36:38.489961 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:38.489939 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a58c255_8ae3_46f3_977e_310e0db54caa.slice/crio-7ad297794852e8e5de302c4c46cadab94b68a094737805e60ae453a766b2a915 WatchSource:0}: Error finding container 7ad297794852e8e5de302c4c46cadab94b68a094737805e60ae453a766b2a915: Status 404 returned error can't find the container with id 7ad297794852e8e5de302c4c46cadab94b68a094737805e60ae453a766b2a915 Jul 16 15:36:38.576404 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.576367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t99dx" event={"ID":"1a58c255-8ae3-46f3-977e-310e0db54caa","Type":"ContainerStarted","Data":"7ad297794852e8e5de302c4c46cadab94b68a094737805e60ae453a766b2a915"} Jul 16 15:36:38.577435 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.577399 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfqqs" event={"ID":"5065548b-266f-4194-b8fe-f44fb431cdd4","Type":"ContainerStarted","Data":"f9535d09bca9571e00287fbadc205c0fcff1d507581d3ba1bbc7b932818b4378"} Jul 16 15:36:38.578428 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.578409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tr9ts" event={"ID":"0e5984f0-0dcb-467c-876d-84c7943edc28","Type":"ContainerStarted","Data":"c596302d26cf01f3e54500666bb00bb00eb2cdb94310c2286fe1a8fc334c1aef"} Jul 16 15:36:38.980049 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:38.980017 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-jslpn"] Jul 16 15:36:39.001839 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.001814 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-jslpn"] Jul 16 15:36:39.001964 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.001912 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.005306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.005167 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 15:36:39.005434 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.005329 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-r4c8g\"" Jul 16 15:36:39.005702 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.005683 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 15:36:39.005758 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.005684 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 15:36:39.006406 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.006393 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 15:36:39.006459 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.006447 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 15:36:39.036563 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.036536 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f399840-a744-424e-88b5-43feb12b4f8f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.036672 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.036589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f399840-a744-424e-88b5-43feb12b4f8f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.036672 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.036636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncd9v\" (UniqueName: \"kubernetes.io/projected/4f399840-a744-424e-88b5-43feb12b4f8f-kube-api-access-ncd9v\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.036744 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.036674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f399840-a744-424e-88b5-43feb12b4f8f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.136937 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.136905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f399840-a744-424e-88b5-43feb12b4f8f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.137106 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.136961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f399840-a744-424e-88b5-43feb12b4f8f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.137106 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.136984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd9v\" (UniqueName: \"kubernetes.io/projected/4f399840-a744-424e-88b5-43feb12b4f8f-kube-api-access-ncd9v\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.137208 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.137160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f399840-a744-424e-88b5-43feb12b4f8f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.137812 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.137790 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f399840-a744-424e-88b5-43feb12b4f8f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.140217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.140199 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f399840-a744-424e-88b5-43feb12b4f8f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.140269 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.140214 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f399840-a744-424e-88b5-43feb12b4f8f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.146654 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.146632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncd9v\" (UniqueName: \"kubernetes.io/projected/4f399840-a744-424e-88b5-43feb12b4f8f-kube-api-access-ncd9v\") pod \"prometheus-operator-64785b6675-jslpn\" (UID: \"4f399840-a744-424e-88b5-43feb12b4f8f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.311244 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.311197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" Jul 16 15:36:39.378016 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.377986 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:39.378212 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.378097 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:39.378720 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.378673 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:39.383485 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.382737 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfqhq\"" Jul 16 15:36:39.383485 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.382959 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 15:36:39.383485 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.383159 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 15:36:39.383485 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.383353 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 15:36:39.384773 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.383535 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2scpx\"" Jul 16 15:36:39.394628 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.389799 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 15:36:39.493364 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.493285 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-jslpn"] Jul 16 15:36:39.497145 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:39.497113 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f399840_a744_424e_88b5_43feb12b4f8f.slice/crio-d85f26d20d3c31bffcbd87076d4747a7db9c6d6e23c37454f45f7ba71a8f6080 WatchSource:0}: Error finding container d85f26d20d3c31bffcbd87076d4747a7db9c6d6e23c37454f45f7ba71a8f6080: Status 404 returned error can't find the container with id d85f26d20d3c31bffcbd87076d4747a7db9c6d6e23c37454f45f7ba71a8f6080 Jul 16 15:36:39.586129 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:39.586036 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" event={"ID":"4f399840-a744-424e-88b5-43feb12b4f8f","Type":"ContainerStarted","Data":"d85f26d20d3c31bffcbd87076d4747a7db9c6d6e23c37454f45f7ba71a8f6080"} Jul 16 15:36:42.596020 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:42.595821 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t99dx" event={"ID":"1a58c255-8ae3-46f3-977e-310e0db54caa","Type":"ContainerStarted","Data":"315249bc6080d15865e5e9e01837de8ce17636e96cb4076ea5016766282e4f3b"} Jul 16 15:36:42.597916 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:42.597889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tr9ts" event={"ID":"0e5984f0-0dcb-467c-876d-84c7943edc28","Type":"ContainerStarted","Data":"aebd24f800f97b3303a18585b31bc3b07f4e63b1975ab9d163e045c207a514f9"} Jul 16 15:36:42.598071 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:42.597939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tr9ts" event={"ID":"0e5984f0-0dcb-467c-876d-84c7943edc28","Type":"ContainerStarted","Data":"d269aa15d7058f0ee4a63408c702478419e92626c5147a7d312970d1b694a1e6"} Jul 16 15:36:42.598071 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:42.598048 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:42.614025 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:42.613962 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t99dx" podStartSLOduration=2.312064728 podStartE2EDuration="5.613941216s" totalImagesPullingTime="3.301876488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:42.612685259 +0000 UTC m=+57.051212087" watchObservedRunningTime="2026-07-16 15:36:42.613941216 +0000 UTC m=+57.052468039" Jul 16 15:36:42.632973 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:42.632914 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tr9ts" podStartSLOduration=2.294046192 podStartE2EDuration="5.632893664s" totalImagesPullingTime="3.338847472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:42.631193068 +0000 UTC m=+57.069719893" watchObservedRunningTime="2026-07-16 15:36:42.632893664 +0000 UTC m=+57.071420487" Jul 16 15:36:43.057229 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.057192 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cdb6b68c4-m7dcm"] Jul 16 15:36:43.071219 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.071185 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdb6b68c4-m7dcm"] Jul 16 15:36:43.071378 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.071192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.074081 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.074056 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 15:36:43.074301 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.074288 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 15:36:43.074468 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.074451 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 15:36:43.074842 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.074821 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 15:36:43.075492 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.075469 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hr47v\"" Jul 16 15:36:43.075739 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.075519 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 15:36:43.075739 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.075700 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 15:36:43.076283 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.076014 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 15:36:43.084059 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.083884 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 15:36:43.170788 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.170715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-service-ca\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.170788 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.170784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-config\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.170997 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.170883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-oauth-config\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.170997 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.170909 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-oauth-serving-cert\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.170997 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.170927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls8x6\" (UniqueName: \"kubernetes.io/projected/a1fc9f6f-e597-41a6-96ba-7c5226064e85-kube-api-access-ls8x6\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.170997 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.170956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-trusted-ca-bundle\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.171165 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.171033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-serving-cert\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.271876 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.271844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-config\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.271989 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.271926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-oauth-config\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.271989 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.271955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-oauth-serving-cert\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.271989 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.271986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8x6\" (UniqueName: \"kubernetes.io/projected/a1fc9f6f-e597-41a6-96ba-7c5226064e85-kube-api-access-ls8x6\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.272150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.272017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-trusted-ca-bundle\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.272150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.272057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-serving-cert\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.272150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.272087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-service-ca\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.272696 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.272602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-config\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.272696 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.272685 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-oauth-serving-cert\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.272872 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.272708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-service-ca\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.273815 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.273771 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-trusted-ca-bundle\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.276780 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.276744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-oauth-config\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.276870 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.276851 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-serving-cert\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.281416 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.281396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8x6\" (UniqueName: \"kubernetes.io/projected/a1fc9f6f-e597-41a6-96ba-7c5226064e85-kube-api-access-ls8x6\") pod \"console-6cdb6b68c4-m7dcm\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.386975 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.386934 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:43.593294 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.593261 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdb6b68c4-m7dcm"] Jul 16 15:36:43.596969 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:43.596936 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fc9f6f_e597_41a6_96ba_7c5226064e85.slice/crio-42a8c9c79602d8ab046be1a7cb69c87cd6b5ef8d3ec6cde64eb99c3690348cc7 WatchSource:0}: Error finding container 42a8c9c79602d8ab046be1a7cb69c87cd6b5ef8d3ec6cde64eb99c3690348cc7: Status 404 returned error can't find the container with id 42a8c9c79602d8ab046be1a7cb69c87cd6b5ef8d3ec6cde64eb99c3690348cc7 Jul 16 15:36:43.604010 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.603963 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdb6b68c4-m7dcm" event={"ID":"a1fc9f6f-e597-41a6-96ba-7c5226064e85","Type":"ContainerStarted","Data":"42a8c9c79602d8ab046be1a7cb69c87cd6b5ef8d3ec6cde64eb99c3690348cc7"} Jul 16 15:36:43.606399 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.606365 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" event={"ID":"4f399840-a744-424e-88b5-43feb12b4f8f","Type":"ContainerStarted","Data":"a7cdabbd9e342b7bdfb22ad740736953ab844dd66d50e591b9943ac3737549df"} Jul 16 15:36:43.606500 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.606428 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" event={"ID":"4f399840-a744-424e-88b5-43feb12b4f8f","Type":"ContainerStarted","Data":"194dc4c7aaf444148a376df03a0aa2d46def7c869d43e76ee811646e01c616cb"} Jul 16 15:36:43.625555 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:43.625510 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-jslpn" podStartSLOduration=1.886132699 podStartE2EDuration="5.625488653s" totalImagesPullingTime="3.739355954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:43.625110511 +0000 UTC m=+58.063637337" watchObservedRunningTime="2026-07-16 15:36:43.625488653 +0000 UTC m=+58.064015477" Jul 16 15:36:45.311880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.311848 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42"] Jul 16 15:36:45.333204 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.333171 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-786zz"] Jul 16 15:36:45.333354 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.333307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.335791 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.335768 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 15:36:45.336229 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.336207 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gpm8f\"" Jul 16 15:36:45.336347 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.336232 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 15:36:45.336813 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.336787 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-qzr2s"] Jul 16 15:36:45.336969 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.336916 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.339397 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.339369 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 15:36:45.339728 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.339629 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 15:36:45.339728 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.339663 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42"] Jul 16 15:36:45.339883 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.339791 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.339883 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.339666 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 15:36:45.339975 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.339904 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4qtrk\"" Jul 16 15:36:45.344843 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.344820 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 15:36:45.345732 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.345711 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-qzr2s"] Jul 16 15:36:45.346168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.346145 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 15:36:45.348007 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.346405 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 15:36:45.348473 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.348374 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7hdsl\"" Jul 16 15:36:45.391346 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.391507 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzlc2\" (UniqueName: \"kubernetes.io/projected/9b02f526-ec6f-4404-96fe-f98f1a013a54-kube-api-access-nzlc2\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.391507 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-sys\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.391507 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.391507 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njp9r\" (UniqueName: \"kubernetes.io/projected/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-kube-api-access-njp9r\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.391748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.391748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.391748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.391748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.391748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-root\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.391748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-wtmp\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.391748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391685 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.391748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.392148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54xw\" (UniqueName: \"kubernetes.io/projected/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-api-access-z54xw\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.392148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.392148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.392148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-tls\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.392148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b02f526-ec6f-4404-96fe-f98f1a013a54-metrics-client-ca\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.392148 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.391925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-textfile\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.492819 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.492819 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492818 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-root\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-wtmp\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492922 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-root\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54xw\" (UniqueName: \"kubernetes.io/projected/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-api-access-z54xw\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.492998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493028 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-tls\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-wtmp\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493058 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b02f526-ec6f-4404-96fe-f98f1a013a54-metrics-client-ca\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-textfile\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzlc2\" (UniqueName: \"kubernetes.io/projected/9b02f526-ec6f-4404-96fe-f98f1a013a54-kube-api-access-nzlc2\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-sys\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493195 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njp9r\" (UniqueName: \"kubernetes.io/projected/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-kube-api-access-njp9r\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493249 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b02f526-ec6f-4404-96fe-f98f1a013a54-sys\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.493607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.494132 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:45.493683 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 15:36:45.494132 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:45.493741 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-tls podName:9b02f526-ec6f-4404-96fe-f98f1a013a54 nodeName:}" failed. No retries permitted until 2026-07-16 15:36:45.993722599 +0000 UTC m=+60.432249409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-tls") pod "node-exporter-786zz" (UID: "9b02f526-ec6f-4404-96fe-f98f1a013a54") : secret "node-exporter-tls" not found Jul 16 15:36:45.494132 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.493813 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-textfile\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.494132 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.494088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.494132 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.494101 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.494132 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.494108 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.494437 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.494288 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b02f526-ec6f-4404-96fe-f98f1a013a54-metrics-client-ca\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.496346 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.496300 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.496557 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.496530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.497006 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.496982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.497165 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.497143 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.497250 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.497217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.503299 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.503265 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54xw\" (UniqueName: \"kubernetes.io/projected/4f12b5ca-d7d4-4042-bf06-fe8b14625a8a-kube-api-access-z54xw\") pod \"kube-state-metrics-785778ff74-qzr2s\" (UID: \"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.503673 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.503651 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzlc2\" (UniqueName: \"kubernetes.io/projected/9b02f526-ec6f-4404-96fe-f98f1a013a54-kube-api-access-nzlc2\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:45.503994 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.503973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njp9r\" (UniqueName: \"kubernetes.io/projected/32bb372b-72c8-416e-b667-d6e8e1b3d9fb-kube-api-access-njp9r\") pod \"openshift-state-metrics-84bd68b896-kjz42\" (UID: \"32bb372b-72c8-416e-b667-d6e8e1b3d9fb\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.525581 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.525558 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hprmr" Jul 16 15:36:45.648403 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.648327 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" Jul 16 15:36:45.664364 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.664347 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" Jul 16 15:36:45.998986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:45.998905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-tls\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:46.001635 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.001588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b02f526-ec6f-4404-96fe-f98f1a013a54-node-exporter-tls\") pod \"node-exporter-786zz\" (UID: \"9b02f526-ec6f-4404-96fe-f98f1a013a54\") " pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:46.259887 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.259863 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4qtrk\"" Jul 16 15:36:46.267887 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.267853 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-786zz" Jul 16 15:36:46.411550 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.411492 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:36:46.414565 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.414534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.420262 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.420237 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mvzws\"" Jul 16 15:36:46.422837 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422122 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 15:36:46.422837 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422218 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 15:36:46.422837 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422332 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 15:36:46.422837 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422438 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 15:36:46.422837 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422533 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 15:36:46.422837 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422128 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 15:36:46.422837 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422823 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 15:36:46.423256 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422858 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 15:36:46.423256 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.422918 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 15:36:46.433996 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.433975 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:36:46.503381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-config-out\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503387 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjnf\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-kube-api-access-jvjnf\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-web-config\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503557 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503967 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-config-volume\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503967 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.503967 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.503808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605223 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjnf\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-kube-api-access-jvjnf\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605335 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605335 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605335 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-web-config\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605502 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605502 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605502 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-config-volume\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605502 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605723 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605723 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-config-out\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605723 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605723 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.605723 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.605702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.606774 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.606529 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.607388 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.607144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.608779 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.608747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.610024 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.609936 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.610024 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.609975 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.610558 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.610200 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.610558 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.610357 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-config-volume\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.610558 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.610474 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.611147 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.611104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-web-config\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.611418 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.611375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-config-out\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.612004 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.611964 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.612168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.612141 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.621342 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.621179 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfqqs" event={"ID":"5065548b-266f-4194-b8fe-f44fb431cdd4","Type":"ContainerStarted","Data":"08f820850976e64c99b2ce3141bd720c9af1a9dfdd715d734516da66bb847984"} Jul 16 15:36:46.624548 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.624177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-786zz" event={"ID":"9b02f526-ec6f-4404-96fe-f98f1a013a54","Type":"ContainerStarted","Data":"9b44db58094d4f61f693e0c2a7f2d3aa44163a78b3427edac9036ce274105e59"} Jul 16 15:36:46.635660 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.635636 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjnf\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-kube-api-access-jvjnf\") pod \"alertmanager-main-0\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.638493 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.638472 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42"] Jul 16 15:36:46.641798 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:46.641777 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bb372b_72c8_416e_b667_d6e8e1b3d9fb.slice/crio-fca921b18e36c17956c16209b5768d2c8dd2117c9027265448f1a6710682db1e WatchSource:0}: Error finding container fca921b18e36c17956c16209b5768d2c8dd2117c9027265448f1a6710682db1e: Status 404 returned error can't find the container with id fca921b18e36c17956c16209b5768d2c8dd2117c9027265448f1a6710682db1e Jul 16 15:36:46.650216 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.650149 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-qzr2s"] Jul 16 15:36:46.654061 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:46.654026 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f12b5ca_d7d4_4042_bf06_fe8b14625a8a.slice/crio-9826a144bc431f4c4cdace9fcc80b50b56f01a26f9348ba46e66be2b64bcfb22 WatchSource:0}: Error finding container 9826a144bc431f4c4cdace9fcc80b50b56f01a26f9348ba46e66be2b64bcfb22: Status 404 returned error can't find the container with id 9826a144bc431f4c4cdace9fcc80b50b56f01a26f9348ba46e66be2b64bcfb22 Jul 16 15:36:46.729078 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.729051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:36:46.921441 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:46.921399 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:36:46.925572 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:46.925547 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84211696_2c80_476e_9528_c09a71aa82de.slice/crio-b2aee9dba9676ef8e19e14de29cf5ae1a1884ce99fe9f02d1c3956bc58a82036 WatchSource:0}: Error finding container b2aee9dba9676ef8e19e14de29cf5ae1a1884ce99fe9f02d1c3956bc58a82036: Status 404 returned error can't find the container with id b2aee9dba9676ef8e19e14de29cf5ae1a1884ce99fe9f02d1c3956bc58a82036 Jul 16 15:36:47.409103 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.409066 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d844dddbc-x58qz"] Jul 16 15:36:47.412656 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.412445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.416187 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.416164 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 16 15:36:47.416400 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.416383 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 16 15:36:47.417319 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.417147 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 16 15:36:47.417319 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.417173 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-txn4k\"" Jul 16 15:36:47.420212 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.417882 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 16 15:36:47.422568 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.420717 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-etg4l6vv1rt7d\"" Jul 16 15:36:47.422568 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.421108 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 16 15:36:47.440047 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.439897 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d844dddbc-x58qz"] Jul 16 15:36:47.512985 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.512950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.513133 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.513004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.513133 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.513050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.513133 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.513084 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf244\" (UniqueName: \"kubernetes.io/projected/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-kube-api-access-xf244\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.513133 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.513118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.513356 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.513204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-metrics-client-ca\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.513356 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.513229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-grpc-tls\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.513356 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.513278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-tls\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.614240 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.614198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf244\" (UniqueName: \"kubernetes.io/projected/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-kube-api-access-xf244\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.614414 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.614259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.614414 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.614341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-metrics-client-ca\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.614414 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.614369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-grpc-tls\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.614573 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.614420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-tls\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.614573 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.614452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.614573 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.614488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.614573 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.614537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.616578 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.616539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-metrics-client-ca\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.619153 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.619124 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-tls\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.619697 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.619632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-grpc-tls\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.620268 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.620200 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.620705 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.620660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.620855 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.620805 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.621674 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.621606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.629598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.629554 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf244\" (UniqueName: \"kubernetes.io/projected/3346f6e5-0fbd-4171-a070-bbd198f9d4f9-kube-api-access-xf244\") pod \"thanos-querier-6d844dddbc-x58qz\" (UID: \"3346f6e5-0fbd-4171-a070-bbd198f9d4f9\") " pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:47.631258 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.631227 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" event={"ID":"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a","Type":"ContainerStarted","Data":"9826a144bc431f4c4cdace9fcc80b50b56f01a26f9348ba46e66be2b64bcfb22"} Jul 16 15:36:47.633536 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.633510 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" event={"ID":"32bb372b-72c8-416e-b667-d6e8e1b3d9fb","Type":"ContainerStarted","Data":"11757b81e51dea5683129da771053a47bf34439047d9e6c8d662429cdd6ca8a0"} Jul 16 15:36:47.633660 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.633568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" event={"ID":"32bb372b-72c8-416e-b667-d6e8e1b3d9fb","Type":"ContainerStarted","Data":"f8dda2621317b6093eb6a44439c001734dfe51b2f9c692f13909a931a53e84f1"} Jul 16 15:36:47.633660 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.633598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" event={"ID":"32bb372b-72c8-416e-b667-d6e8e1b3d9fb","Type":"ContainerStarted","Data":"fca921b18e36c17956c16209b5768d2c8dd2117c9027265448f1a6710682db1e"} Jul 16 15:36:47.635530 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.635507 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerStarted","Data":"b2aee9dba9676ef8e19e14de29cf5ae1a1884ce99fe9f02d1c3956bc58a82036"} Jul 16 15:36:47.737922 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:47.737832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:36:49.655583 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.655549 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8468d64cff-92r2h"] Jul 16 15:36:49.672478 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.672443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.676110 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.675916 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 15:36:49.677075 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.677053 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-acdgis1uddsie\"" Jul 16 15:36:49.678071 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.677644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 15:36:49.678071 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.677877 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pfg29\"" Jul 16 15:36:49.678257 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.678079 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 15:36:49.678306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.678293 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 15:36:49.679444 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.679376 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8468d64cff-92r2h"] Jul 16 15:36:49.736538 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.736497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbcf\" (UniqueName: \"kubernetes.io/projected/f6a9b491-69ae-4979-a4f7-7f62bd49043b-kube-api-access-fdbcf\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.736538 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.736540 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-client-ca-bundle\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.736808 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.736627 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-secret-metrics-server-tls\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.736808 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.736652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9b491-69ae-4979-a4f7-7f62bd49043b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.736808 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.736677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-secret-metrics-server-client-certs\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.736808 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.736725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6a9b491-69ae-4979-a4f7-7f62bd49043b-metrics-server-audit-profiles\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.736808 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.736803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6a9b491-69ae-4979-a4f7-7f62bd49043b-audit-log\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.837905 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.837870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-secret-metrics-server-tls\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.838060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.837918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9b491-69ae-4979-a4f7-7f62bd49043b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.838060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.837988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-secret-metrics-server-client-certs\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.838060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.838039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6a9b491-69ae-4979-a4f7-7f62bd49043b-metrics-server-audit-profiles\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.838266 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.838245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6a9b491-69ae-4979-a4f7-7f62bd49043b-audit-log\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.838435 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.838333 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbcf\" (UniqueName: \"kubernetes.io/projected/f6a9b491-69ae-4979-a4f7-7f62bd49043b-kube-api-access-fdbcf\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.838435 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.838372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-client-ca-bundle\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.838626 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.838537 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9b491-69ae-4979-a4f7-7f62bd49043b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.838762 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.838734 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6a9b491-69ae-4979-a4f7-7f62bd49043b-audit-log\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.839071 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.839047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6a9b491-69ae-4979-a4f7-7f62bd49043b-metrics-server-audit-profiles\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.840575 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.840556 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-secret-metrics-server-tls\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.840821 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.840801 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-secret-metrics-server-client-certs\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.840886 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.840816 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6a9b491-69ae-4979-a4f7-7f62bd49043b-client-ca-bundle\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.850385 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.850359 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbcf\" (UniqueName: \"kubernetes.io/projected/f6a9b491-69ae-4979-a4f7-7f62bd49043b-kube-api-access-fdbcf\") pod \"metrics-server-8468d64cff-92r2h\" (UID: \"f6a9b491-69ae-4979-a4f7-7f62bd49043b\") " pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:49.986349 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:49.986237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:36:50.153203 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.153170 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp"] Jul 16 15:36:50.156559 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.156534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:50.161349 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.161186 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qjvcb\"" Jul 16 15:36:50.161349 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.161186 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 15:36:50.172870 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.172798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp"] Jul 16 15:36:50.242480 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.242389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vm9l\" (UniqueName: \"kubernetes.io/projected/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-kube-api-access-6vm9l\") pod \"monitoring-plugin-5f68df7f58-svnwp\" (UID: \"4eaa8f7c-4a05-4758-85f7-88d84d55b7af\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:50.242661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.242543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-svnwp\" (UID: \"4eaa8f7c-4a05-4758-85f7-88d84d55b7af\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:50.343501 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.343469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vm9l\" (UniqueName: \"kubernetes.io/projected/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-kube-api-access-6vm9l\") pod \"monitoring-plugin-5f68df7f58-svnwp\" (UID: \"4eaa8f7c-4a05-4758-85f7-88d84d55b7af\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:50.343677 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.343600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-svnwp\" (UID: \"4eaa8f7c-4a05-4758-85f7-88d84d55b7af\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:50.343757 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:50.343742 2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 16 15:36:50.343863 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:36:50.343850 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-monitoring-plugin-cert podName:4eaa8f7c-4a05-4758-85f7-88d84d55b7af nodeName:}" failed. No retries permitted until 2026-07-16 15:36:50.843832911 +0000 UTC m=+65.282359712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-svnwp" (UID: "4eaa8f7c-4a05-4758-85f7-88d84d55b7af") : secret "monitoring-plugin-cert" not found Jul 16 15:36:50.356079 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.356051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vm9l\" (UniqueName: \"kubernetes.io/projected/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-kube-api-access-6vm9l\") pod \"monitoring-plugin-5f68df7f58-svnwp\" (UID: \"4eaa8f7c-4a05-4758-85f7-88d84d55b7af\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:50.571903 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.571847 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d7c749d-xjlfg"] Jul 16 15:36:50.582698 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.582662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.595444 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.595415 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7554f75ffc-dckgm"] Jul 16 15:36:50.598336 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.598316 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d7c749d-xjlfg"] Jul 16 15:36:50.598484 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.598423 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.607510 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.607486 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 15:36:50.607766 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.607737 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 15:36:50.608812 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.608790 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rcfnb\"" Jul 16 15:36:50.608812 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.608808 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 15:36:50.609074 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.609058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 15:36:50.609390 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.609371 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 15:36:50.615270 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.615249 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 15:36:50.634938 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.634909 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7554f75ffc-dckgm"] Jul 16 15:36:50.646253 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-serving-cert\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.646370 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8qtd\" (UniqueName: \"kubernetes.io/projected/da234646-0f6e-465e-b288-0e7c5a9ef002-kube-api-access-h8qtd\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.646431 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-federate-client-tls\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.646486 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-service-ca\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.646546 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-console-config\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.646546 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-telemeter-client-tls\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.646655 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.646725 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-metrics-client-ca\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.646836 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646737 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.646836 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646767 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-trusted-ca-bundle\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.646836 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-oauth-serving-cert\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.646955 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-oauth-config\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.646955 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-serving-certs-ca-bundle\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.647044 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.646975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-secret-telemeter-client\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.647083 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.647039 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68h2x\" (UniqueName: \"kubernetes.io/projected/656dcbdb-23de-4e81-a62f-49552fb6df29-kube-api-access-68h2x\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.747732 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.747703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-serving-cert\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.747751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8qtd\" (UniqueName: \"kubernetes.io/projected/da234646-0f6e-465e-b288-0e7c5a9ef002-kube-api-access-h8qtd\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.747778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-federate-client-tls\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.747807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-service-ca\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.747832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-console-config\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.747900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-telemeter-client-tls\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.747940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.747999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-metrics-client-ca\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.748026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.748095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.748062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-trusted-ca-bundle\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.748535 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.748118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-oauth-serving-cert\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.748535 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.748173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-oauth-config\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.748535 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.748204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-serving-certs-ca-bundle\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.748535 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.748294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-secret-telemeter-client\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.748535 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.748339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68h2x\" (UniqueName: \"kubernetes.io/projected/656dcbdb-23de-4e81-a62f-49552fb6df29-kube-api-access-68h2x\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.748795 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.748601 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-console-config\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.749030 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.749006 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.749314 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.749287 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-serving-certs-ca-bundle\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.749824 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.749798 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/656dcbdb-23de-4e81-a62f-49552fb6df29-metrics-client-ca\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.750020 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.750002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-service-ca\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.750431 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.750388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-oauth-serving-cert\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.750573 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.750550 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-trusted-ca-bundle\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.750816 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.750774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-serving-cert\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.751420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.751396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-oauth-config\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.751694 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.751679 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-telemeter-client-tls\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.751867 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.751841 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-secret-telemeter-client\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.752298 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.752258 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.752935 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.752914 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/656dcbdb-23de-4e81-a62f-49552fb6df29-federate-client-tls\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.758952 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.758925 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8qtd\" (UniqueName: \"kubernetes.io/projected/da234646-0f6e-465e-b288-0e7c5a9ef002-kube-api-access-h8qtd\") pod \"console-74d7c749d-xjlfg\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.759080 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.759061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68h2x\" (UniqueName: \"kubernetes.io/projected/656dcbdb-23de-4e81-a62f-49552fb6df29-kube-api-access-68h2x\") pod \"telemeter-client-7554f75ffc-dckgm\" (UID: \"656dcbdb-23de-4e81-a62f-49552fb6df29\") " pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:50.850020 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.849948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-svnwp\" (UID: \"4eaa8f7c-4a05-4758-85f7-88d84d55b7af\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:50.852750 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.852723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4eaa8f7c-4a05-4758-85f7-88d84d55b7af-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-svnwp\" (UID: \"4eaa8f7c-4a05-4758-85f7-88d84d55b7af\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:50.892886 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.892854 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:36:50.908220 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:50.908200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" Jul 16 15:36:51.068364 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.068329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:36:51.217641 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.217497 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d844dddbc-x58qz"] Jul 16 15:36:51.608862 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:51.608827 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3346f6e5_0fbd_4171_a070_bbd198f9d4f9.slice/crio-165a95c6d659f115d05f9ccadc9b742b8a8adb16b11aedfacf191a20f151ff30 WatchSource:0}: Error finding container 165a95c6d659f115d05f9ccadc9b742b8a8adb16b11aedfacf191a20f151ff30: Status 404 returned error can't find the container with id 165a95c6d659f115d05f9ccadc9b742b8a8adb16b11aedfacf191a20f151ff30 Jul 16 15:36:51.651765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.651733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" event={"ID":"3346f6e5-0fbd-4171-a070-bbd198f9d4f9","Type":"ContainerStarted","Data":"165a95c6d659f115d05f9ccadc9b742b8a8adb16b11aedfacf191a20f151ff30"} Jul 16 15:36:51.764152 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.763408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:51.774328 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.767964 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 15:36:51.784014 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.783946 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8921e8d-2506-4ff1-abc3-21abb1038d4c-metrics-certs\") pod \"network-metrics-daemon-nz489\" (UID: \"b8921e8d-2506-4ff1-abc3-21abb1038d4c\") " pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:51.864989 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.864483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:51.868590 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.868504 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8468d64cff-92r2h"] Jul 16 15:36:51.873793 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.873586 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 15:36:51.888109 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.885588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7962491-319d-4d1c-8d70-74706e135117-original-pull-secret\") pod \"global-pull-secret-syncer-sdfct\" (UID: \"e7962491-319d-4d1c-8d70-74706e135117\") " pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:51.912974 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.912925 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp"] Jul 16 15:36:51.965087 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.965044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:51.973390 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:51.973366 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaa8f7c_4a05_4758_85f7_88d84d55b7af.slice/crio-229406010fc9483a1f4c981b1a31d6801b89ab1a6141209d85ff95f0139f34d5 WatchSource:0}: Error finding container 229406010fc9483a1f4c981b1a31d6801b89ab1a6141209d85ff95f0139f34d5: Status 404 returned error can't find the container with id 229406010fc9483a1f4c981b1a31d6801b89ab1a6141209d85ff95f0139f34d5 Jul 16 15:36:51.974197 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.974174 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 15:36:51.979215 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.979189 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 15:36:51.988842 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:51.988789 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8fpb\" (UniqueName: \"kubernetes.io/projected/bea76fa1-573a-410a-958d-b63840be1aed-kube-api-access-z8fpb\") pod \"network-check-target-8k54x\" (UID: \"bea76fa1-573a-410a-958d-b63840be1aed\") " pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:52.008705 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.008475 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfqhq\"" Jul 16 15:36:52.016646 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.013998 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d7c749d-xjlfg"] Jul 16 15:36:52.017942 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.017670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nz489" Jul 16 15:36:52.024434 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.024412 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdfct" Jul 16 15:36:52.026889 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:52.025483 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda234646_0f6e_465e_b288_0e7c5a9ef002.slice/crio-28a8e2119597b3982ccd75eafb4e7fa3d4cddc80d54976aa91e7f4c91bf28554 WatchSource:0}: Error finding container 28a8e2119597b3982ccd75eafb4e7fa3d4cddc80d54976aa91e7f4c91bf28554: Status 404 returned error can't find the container with id 28a8e2119597b3982ccd75eafb4e7fa3d4cddc80d54976aa91e7f4c91bf28554 Jul 16 15:36:52.027534 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.027402 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7554f75ffc-dckgm"] Jul 16 15:36:52.036974 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.036922 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2scpx\"" Jul 16 15:36:52.037334 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.037315 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:36:52.113495 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.112573 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:36:52.143821 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.142228 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.215909 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.215201 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 15:36:52.215909 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.215525 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.218276 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.218517 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.218725 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cvc984r5dp3hi\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.218909 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.219057 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.219086 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.219182 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.219264 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.219420 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.219603 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 15:36:52.220420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.220250 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 15:36:52.222258 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.222240 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5s8ks\"" Jul 16 15:36:52.222743 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.222727 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 15:36:52.230437 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.230419 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 15:36:52.272549 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.272296 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 15:36:52.273352 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.273313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.273712 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.273690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.273951 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.273933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.274134 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.274109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.274342 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.274324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.274485 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.274470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.274758 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.274741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.275157 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.275086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.275319 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.275117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.275552 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.275460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.275732 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.275689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.275910 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.275836 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.276065 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.275870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.276242 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.276200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.276434 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.276371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.276604 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.276546 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.276860 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.276808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.277063 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.276978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.277211 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.277009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chw5g\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-kube-api-access-chw5g\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.277447 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.277354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.334598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.334425 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nz489"] Jul 16 15:36:52.344715 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:52.344682 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8921e8d_2506_4ff1_abc3_21abb1038d4c.slice/crio-7acf3daadf0cacd820329ff5dcea9131e54dc6dbd27bdc42a9cb3e7ba48d5134 WatchSource:0}: Error finding container 7acf3daadf0cacd820329ff5dcea9131e54dc6dbd27bdc42a9cb3e7ba48d5134: Status 404 returned error can't find the container with id 7acf3daadf0cacd820329ff5dcea9131e54dc6dbd27bdc42a9cb3e7ba48d5134 Jul 16 15:36:52.378915 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.378915 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.378915 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.378915 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.378915 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.378915 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378882 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chw5g\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-kube-api-access-chw5g\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.378985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379009 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.380217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.379435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.381467 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.380099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.381541 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.381518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.383285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.382161 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.383285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.382457 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.383285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.382680 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.383285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.382964 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.383285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.382993 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.383646 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.383408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.384261 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.384236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.386636 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.386575 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.386636 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:36:52.386578 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7962491_319d_4d1c_8d70_74706e135117.slice/crio-54efe7e903bf954f20c8e1a54e583aaabfb13b87ba5ec2f1b7927eec6e236744 WatchSource:0}: Error finding container 54efe7e903bf954f20c8e1a54e583aaabfb13b87ba5ec2f1b7927eec6e236744: Status 404 returned error can't find the container with id 54efe7e903bf954f20c8e1a54e583aaabfb13b87ba5ec2f1b7927eec6e236744 Jul 16 15:36:52.386838 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.386813 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.387910 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.387890 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sdfct"] Jul 16 15:36:52.389712 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.388836 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.389712 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.389166 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.389712 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.389290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.389712 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.389707 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.390858 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.390645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.391162 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.391120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.391436 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.391356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.391982 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.391728 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.407833 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.407780 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chw5g\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-kube-api-access-chw5g\") pod \"prometheus-k8s-0\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.495373 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.495122 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:36:52.538565 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.538520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8k54x"] Jul 16 15:36:52.610123 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.609185 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tr9ts" Jul 16 15:36:52.662419 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.661527 2573 generic.go:356] "Generic (PLEG): container finished" podID="9b02f526-ec6f-4404-96fe-f98f1a013a54" containerID="40013faef7a1827a941f22adc6ab35828211dfcd5662fc6272c5938d0e814244" exitCode=0 Jul 16 15:36:52.662419 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.661634 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-786zz" event={"ID":"9b02f526-ec6f-4404-96fe-f98f1a013a54","Type":"ContainerDied","Data":"40013faef7a1827a941f22adc6ab35828211dfcd5662fc6272c5938d0e814244"} Jul 16 15:36:52.670901 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.670870 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" event={"ID":"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a","Type":"ContainerStarted","Data":"bd786304d671070972dd5deec00c89be8dff291ba59f91f104565d53235f05de"} Jul 16 15:36:52.671041 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.670938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" event={"ID":"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a","Type":"ContainerStarted","Data":"95c832175410f8ea5949fc98e26726cc6bc0becef196e919b3437569c919d8bc"} Jul 16 15:36:52.671041 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.670965 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" event={"ID":"4f12b5ca-d7d4-4042-bf06-fe8b14625a8a","Type":"ContainerStarted","Data":"10dcffdc4c1d07ade9cc1bcffd556772c2ea06df7246ce92329196560507f5cc"} Jul 16 15:36:52.693199 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.693170 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" event={"ID":"32bb372b-72c8-416e-b667-d6e8e1b3d9fb","Type":"ContainerStarted","Data":"70671480c728e5261b828a00e45e10a8b1c46405f375811d5e2ed8401db5793c"} Jul 16 15:36:52.695884 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.695856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8k54x" event={"ID":"bea76fa1-573a-410a-958d-b63840be1aed","Type":"ContainerStarted","Data":"aa2b9cec77b5d13ad4a3f68d04cd7da62faaa48c584e9ba5883dd2c91e001833"} Jul 16 15:36:52.698479 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.698451 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sdfct" event={"ID":"e7962491-319d-4d1c-8d70-74706e135117","Type":"ContainerStarted","Data":"54efe7e903bf954f20c8e1a54e583aaabfb13b87ba5ec2f1b7927eec6e236744"} Jul 16 15:36:52.710255 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.710212 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nz489" event={"ID":"b8921e8d-2506-4ff1-abc3-21abb1038d4c","Type":"ContainerStarted","Data":"7acf3daadf0cacd820329ff5dcea9131e54dc6dbd27bdc42a9cb3e7ba48d5134"} Jul 16 15:36:52.714200 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.714121 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" event={"ID":"656dcbdb-23de-4e81-a62f-49552fb6df29","Type":"ContainerStarted","Data":"0b83e641a836594cd0da79b7a0254c4c93ade15f798f10583c5caf33902636f7"} Jul 16 15:36:52.715452 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.715433 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:36:52.716817 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.716763 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-qzr2s" podStartSLOduration=2.708736237 podStartE2EDuration="7.716562236s" totalImagesPullingTime="5.007825999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:52.712288424 +0000 UTC m=+67.150815253" watchObservedRunningTime="2026-07-16 15:36:52.716562236 +0000 UTC m=+67.155089057" Jul 16 15:36:52.718638 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.717565 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d7c749d-xjlfg" event={"ID":"da234646-0f6e-465e-b288-0e7c5a9ef002","Type":"ContainerStarted","Data":"3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d"} Jul 16 15:36:52.718638 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.717602 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d7c749d-xjlfg" event={"ID":"da234646-0f6e-465e-b288-0e7c5a9ef002","Type":"ContainerStarted","Data":"28a8e2119597b3982ccd75eafb4e7fa3d4cddc80d54976aa91e7f4c91bf28554"} Jul 16 15:36:52.727109 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.727065 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfqqs" event={"ID":"5065548b-266f-4194-b8fe-f44fb431cdd4","Type":"ContainerStarted","Data":"8b3e4fe5d2e1b6cf7a2298450b63db374f69cd1887202b00a10b595d2186ecc0"} Jul 16 15:36:52.733067 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.733036 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdb6b68c4-m7dcm" event={"ID":"a1fc9f6f-e597-41a6-96ba-7c5226064e85","Type":"ContainerStarted","Data":"9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8"} Jul 16 15:36:52.734937 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.734910 2573 generic.go:356] "Generic (PLEG): container finished" podID="84211696-2c80-476e-9528-c09a71aa82de" containerID="10f11609901add420b7aa96bad939480cec697c6a9e8d394b7cbc0309f342731" exitCode=0 Jul 16 15:36:52.735021 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.734983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerDied","Data":"10f11609901add420b7aa96bad939480cec697c6a9e8d394b7cbc0309f342731"} Jul 16 15:36:52.739788 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.739742 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kjz42" podStartSLOduration=2.840640327 podStartE2EDuration="7.739724327s" totalImagesPullingTime="4.899084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:52.738212513 +0000 UTC m=+67.176739338" watchObservedRunningTime="2026-07-16 15:36:52.739724327 +0000 UTC m=+67.178251161" Jul 16 15:36:52.740530 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.740496 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" event={"ID":"f6a9b491-69ae-4979-a4f7-7f62bd49043b","Type":"ContainerStarted","Data":"f7cb75352bffd59b50f9d8c6b14834655ba34b7a4a5595e88ef38d2b2b88a8b4"} Jul 16 15:36:52.754225 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.754125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" event={"ID":"4eaa8f7c-4a05-4758-85f7-88d84d55b7af","Type":"ContainerStarted","Data":"229406010fc9483a1f4c981b1a31d6801b89ab1a6141209d85ff95f0139f34d5"} Jul 16 15:36:52.764573 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.764403 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cdb6b68c4-m7dcm" podStartSLOduration=2.5531530460000003 podStartE2EDuration="9.764385439s" totalImagesPullingTime="7.211232393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:52.763324398 +0000 UTC m=+67.201851225" watchObservedRunningTime="2026-07-16 15:36:52.764385439 +0000 UTC m=+67.202912268" Jul 16 15:36:52.788717 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:52.786394 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74d7c749d-xjlfg" podStartSLOduration=2.7863759310000003 podStartE2EDuration="2.786375931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:52.786011927 +0000 UTC m=+67.224538751" watchObservedRunningTime="2026-07-16 15:36:52.786375931 +0000 UTC m=+67.224902755" Jul 16 15:36:53.388117 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.387986 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:53.388117 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.388086 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:36:53.389673 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.389330 2573 patch_prober.go:28] interesting pod/console-6cdb6b68c4-m7dcm container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 16 15:36:53.389673 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.389386 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6cdb6b68c4-m7dcm" podUID="a1fc9f6f-e597-41a6-96ba-7c5226064e85" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 16 15:36:53.765686 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.765124 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-786zz" event={"ID":"9b02f526-ec6f-4404-96fe-f98f1a013a54","Type":"ContainerStarted","Data":"9bd729e3873ed283f17bae14d1c13200cae36ca805a027171d30d25fd8e63e4a"} Jul 16 15:36:53.765686 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.765189 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-786zz" event={"ID":"9b02f526-ec6f-4404-96fe-f98f1a013a54","Type":"ContainerStarted","Data":"c4ce0ed471c2ef34e2a3c3947fa79ef9c9092f54c886eacb75e0dce879c87f68"} Jul 16 15:36:53.772646 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.771800 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" exitCode=0 Jul 16 15:36:53.778526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.773282 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerDied","Data":"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367"} Jul 16 15:36:53.778526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.773356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerStarted","Data":"604247fc120b47a6ae94ce325d813e5e7a502c373b2e86397b2cfc590e7aff22"} Jul 16 15:36:53.837012 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:36:53.836587 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-786zz" podStartSLOduration=3.586885685 podStartE2EDuration="8.836564559s" totalImagesPullingTime="5.126857759s" totalInitContainerRuntime="122.821115ms" isStatefulPod=true podCreationTimestamp="2026-07-16 15:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:36:53.801987344 +0000 UTC m=+68.240514169" watchObservedRunningTime="2026-07-16 15:36:53.836564559 +0000 UTC m=+68.275091386" Jul 16 15:37:00.893985 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:00.893930 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:37:00.894441 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:00.894020 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:37:00.899425 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:00.899405 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:37:01.812152 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:01.812117 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:37:01.861363 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:01.861337 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdb6b68c4-m7dcm"] Jul 16 15:37:03.820168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.820128 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" event={"ID":"656dcbdb-23de-4e81-a62f-49552fb6df29","Type":"ContainerStarted","Data":"53e455550a032d797257e8cd02682e7db680f896fb601201f53830eda13c289a"} Jul 16 15:37:03.820667 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.820239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" event={"ID":"656dcbdb-23de-4e81-a62f-49552fb6df29","Type":"ContainerStarted","Data":"3f440661a19bbd71af9dbaebbfeb0ba8cde98f9c04d0f4cb9b254ebe2a03cecc"} Jul 16 15:37:03.820667 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.820270 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" event={"ID":"656dcbdb-23de-4e81-a62f-49552fb6df29","Type":"ContainerStarted","Data":"b3ed7cb2df187834fa63b150d65cbd656433004dbb094f247fae9183404c669d"} Jul 16 15:37:03.823377 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.823351 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerStarted","Data":"80c5c3b51fd8a21a1941ae92e3d0898d35cb72dcdcc9388e4c91aaba22a47fe7"} Jul 16 15:37:03.823503 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.823414 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerStarted","Data":"78da22e57eb0eaaef7b9f018465536b9659b5b9e13f36210802e1489f96778ae"} Jul 16 15:37:03.823503 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.823456 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerStarted","Data":"9b1fca385218b91c608eed554b787d03edd99e06dd6d0680305ec28f951989e2"} Jul 16 15:37:03.823503 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.823493 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerStarted","Data":"5ef55611ab8a89f58a5e76a006cc7c8721bd26c36ae47b781107d5007fdd9f48"} Jul 16 15:37:03.823682 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.823530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerStarted","Data":"89af2e4755b5c4332c60d6d3914bfba4a9a378d07985a488e2dffc69e0ae55f8"} Jul 16 15:37:03.825511 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.825489 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" event={"ID":"3346f6e5-0fbd-4171-a070-bbd198f9d4f9","Type":"ContainerStarted","Data":"9d25075619ee89fd9a17427fd1beb642633f67ad70cd46ca8dea3337b467db70"} Jul 16 15:37:03.825608 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.825547 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" event={"ID":"3346f6e5-0fbd-4171-a070-bbd198f9d4f9","Type":"ContainerStarted","Data":"c99bcfde004b1a79fdc63e9a63dff3288dc1df5b7abe6f30273cae134105120a"} Jul 16 15:37:03.825608 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.825589 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" event={"ID":"3346f6e5-0fbd-4171-a070-bbd198f9d4f9","Type":"ContainerStarted","Data":"3d1e3f0de956ece3643e696a3a0b94f38068aa5a729b3323e7493c88d1cdcfa4"} Jul 16 15:37:03.827105 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.827083 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" event={"ID":"f6a9b491-69ae-4979-a4f7-7f62bd49043b","Type":"ContainerStarted","Data":"d0796d8661c416849030c6475ddf9627a4e8a4064b9e5a0c4bcc79910c93f892"} Jul 16 15:37:03.828821 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.828788 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nz489" event={"ID":"b8921e8d-2506-4ff1-abc3-21abb1038d4c","Type":"ContainerStarted","Data":"1d7e521dcdb9124710968fed850892eb5b142887d6948165f2eaa0acba623323"} Jul 16 15:37:03.828913 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.828831 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nz489" event={"ID":"b8921e8d-2506-4ff1-abc3-21abb1038d4c","Type":"ContainerStarted","Data":"631c601772a8e6db8957e749e185353de61ea54cca6e35a06979b97d2b6a2263"} Jul 16 15:37:03.830277 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.830256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" event={"ID":"4eaa8f7c-4a05-4758-85f7-88d84d55b7af","Type":"ContainerStarted","Data":"1b6717c0b515f0b74ebbcd2ecf71f52a50a6689918feb8cc003ee4c0e98eece0"} Jul 16 15:37:03.830383 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.830298 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:37:03.831805 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.831761 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8k54x" event={"ID":"bea76fa1-573a-410a-958d-b63840be1aed","Type":"ContainerStarted","Data":"be7c86325f22b4307ed1758523d1e6aefece0094b61055ff9a16ce1814476040"} Jul 16 15:37:03.831916 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.831882 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:37:03.834118 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.834094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfqqs" event={"ID":"5065548b-266f-4194-b8fe-f44fb431cdd4","Type":"ContainerStarted","Data":"efd84009400839596f718d6ce872ff069213901887f601d51d2b8f15867f14f1"} Jul 16 15:37:03.835994 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.835976 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" Jul 16 15:37:03.837551 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.837532 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerStarted","Data":"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964"} Jul 16 15:37:03.837664 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.837595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerStarted","Data":"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32"} Jul 16 15:37:03.837709 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.837668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerStarted","Data":"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67"} Jul 16 15:37:03.837759 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.837709 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerStarted","Data":"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7"} Jul 16 15:37:03.837759 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.837741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerStarted","Data":"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c"} Jul 16 15:37:03.837855 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.837764 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerStarted","Data":"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8"} Jul 16 15:37:03.839011 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.838988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sdfct" event={"ID":"e7962491-319d-4d1c-8d70-74706e135117","Type":"ContainerStarted","Data":"f35fa0b9ede4a2c4a3612277054f5b2276b3f67127f727834bb2b68365df80b1"} Jul 16 15:37:03.844558 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.844399 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7554f75ffc-dckgm" podStartSLOduration=2.90922343 podStartE2EDuration="13.844386286s" totalImagesPullingTime="10.935162856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:03.841681823 +0000 UTC m=+78.280208646" watchObservedRunningTime="2026-07-16 15:37:03.844386286 +0000 UTC m=+78.282913105" Jul 16 15:37:03.862409 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.862368 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nz489" podStartSLOduration=67.228869889 podStartE2EDuration="1m17.862356761s" totalImagesPullingTime="10.633486872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:03.861013047 +0000 UTC m=+78.299539869" watchObservedRunningTime="2026-07-16 15:37:03.862356761 +0000 UTC m=+78.300883583" Jul 16 15:37:03.897510 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.897458 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.606370225 podStartE2EDuration="11.897439488s" totalImagesPullingTime="9.254734516s" totalInitContainerRuntime="36.334747ms" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:03.89639761 +0000 UTC m=+78.334924437" watchObservedRunningTime="2026-07-16 15:37:03.897439488 +0000 UTC m=+78.335966311" Jul 16 15:37:03.915359 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.915315 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8k54x" podStartSLOduration=67.48417317 podStartE2EDuration="1m17.915300977s" totalImagesPullingTime="10.431127807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:03.91451563 +0000 UTC m=+78.353042453" watchObservedRunningTime="2026-07-16 15:37:03.915300977 +0000 UTC m=+78.353827800" Jul 16 15:37:03.952122 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.952061 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sdfct" podStartSLOduration=67.319314344 podStartE2EDuration="1m17.952047336s" totalImagesPullingTime="10.632732992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:03.930634123 +0000 UTC m=+78.369160937" watchObservedRunningTime="2026-07-16 15:37:03.952047336 +0000 UTC m=+78.390574192" Jul 16 15:37:03.952972 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.952940 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" podStartSLOduration=3.864154817 podStartE2EDuration="14.952930478s" totalImagesPullingTime="11.088775661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:03.951342147 +0000 UTC m=+78.389868973" watchObservedRunningTime="2026-07-16 15:37:03.952930478 +0000 UTC m=+78.391457301" Jul 16 15:37:03.972189 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.972143 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kfqqs" podStartSLOduration=2.706219535 podStartE2EDuration="26.972130234s" totalImagesPullingTime="24.265910699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 15:36:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:03.970875735 +0000 UTC m=+78.409402559" watchObservedRunningTime="2026-07-16 15:37:03.972130234 +0000 UTC m=+78.410657056" Jul 16 15:37:03.987092 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:03.987046 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-svnwp" podStartSLOduration=4.040949392 podStartE2EDuration="13.987031764s" totalImagesPullingTime="9.946082372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:03.985995819 +0000 UTC m=+78.424522642" watchObservedRunningTime="2026-07-16 15:37:03.987031764 +0000 UTC m=+78.425558586" Jul 16 15:37:05.852445 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:05.852403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerStarted","Data":"05b5fdb006118d1f0fc373119ef8273281a5ca880c54772e86349bb2eb0d99d5"} Jul 16 15:37:05.855245 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:05.855219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" event={"ID":"3346f6e5-0fbd-4171-a070-bbd198f9d4f9","Type":"ContainerStarted","Data":"db05934c429faa1b34a58af94d94c5476a4576726d5aaf8e1b399c0c2ab7ec6c"} Jul 16 15:37:05.855381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:05.855289 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" event={"ID":"3346f6e5-0fbd-4171-a070-bbd198f9d4f9","Type":"ContainerStarted","Data":"9e0fc047fb5969fad18ec68a116c57ba1a7d5b2db12f4f3a7220a3e230402815"} Jul 16 15:37:05.855381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:05.855330 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" event={"ID":"3346f6e5-0fbd-4171-a070-bbd198f9d4f9","Type":"ContainerStarted","Data":"0060b1eaeed5de5b40edf67580193ebb6d6c9f3e669996592f83bea6e13fc174"} Jul 16 15:37:05.885126 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:05.885078 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.464732528 podStartE2EDuration="19.885060084s" totalImagesPullingTime="16.377850624s" totalInitContainerRuntime="42.476932ms" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:05.882632181 +0000 UTC m=+80.321158996" watchObservedRunningTime="2026-07-16 15:37:05.885060084 +0000 UTC m=+80.323586908" Jul 16 15:37:05.907192 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:05.907130 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" podStartSLOduration=5.966110605 podStartE2EDuration="18.907113247s" totalImagesPullingTime="12.941002642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:36:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:37:05.906353619 +0000 UTC m=+80.344880445" watchObservedRunningTime="2026-07-16 15:37:05.907113247 +0000 UTC m=+80.345640071" Jul 16 15:37:06.860364 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:06.860336 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:37:07.495586 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:07.495550 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:37:09.987048 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:09.987014 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:37:09.987569 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:09.987115 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:37:12.869257 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:12.869232 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d844dddbc-x58qz" Jul 16 15:37:15.581373 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:15.581339 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d7c749d-xjlfg"] Jul 16 15:37:26.881589 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:26.881524 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cdb6b68c4-m7dcm" podUID="a1fc9f6f-e597-41a6-96ba-7c5226064e85" containerName="console" containerID="cri-o://9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8" gracePeriod=15 Jul 16 15:37:27.124280 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.124258 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:37:27.234453 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234370 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-oauth-serving-cert\") pod \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " Jul 16 15:37:27.234453 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234406 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-serving-cert\") pod \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " Jul 16 15:37:27.234453 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234451 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-service-ca\") pod \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " Jul 16 15:37:27.234718 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234486 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls8x6\" (UniqueName: \"kubernetes.io/projected/a1fc9f6f-e597-41a6-96ba-7c5226064e85-kube-api-access-ls8x6\") pod \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " Jul 16 15:37:27.234718 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234534 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-trusted-ca-bundle\") pod \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " Jul 16 15:37:27.234718 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234562 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-config\") pod \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " Jul 16 15:37:27.234718 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234599 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-oauth-config\") pod \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\" (UID: \"a1fc9f6f-e597-41a6-96ba-7c5226064e85\") " Jul 16 15:37:27.234929 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234823 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1fc9f6f-e597-41a6-96ba-7c5226064e85" (UID: "a1fc9f6f-e597-41a6-96ba-7c5226064e85"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:37:27.234929 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.234918 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:27.235045 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.235013 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-config" (OuterVolumeSpecName: "console-config") pod "a1fc9f6f-e597-41a6-96ba-7c5226064e85" (UID: "a1fc9f6f-e597-41a6-96ba-7c5226064e85"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:37:27.235163 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.235028 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a1fc9f6f-e597-41a6-96ba-7c5226064e85" (UID: "a1fc9f6f-e597-41a6-96ba-7c5226064e85"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:37:27.235292 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.235218 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a1fc9f6f-e597-41a6-96ba-7c5226064e85" (UID: "a1fc9f6f-e597-41a6-96ba-7c5226064e85"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:37:27.236919 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.236895 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a1fc9f6f-e597-41a6-96ba-7c5226064e85" (UID: "a1fc9f6f-e597-41a6-96ba-7c5226064e85"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:37:27.237126 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.237107 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fc9f6f-e597-41a6-96ba-7c5226064e85-kube-api-access-ls8x6" (OuterVolumeSpecName: "kube-api-access-ls8x6") pod "a1fc9f6f-e597-41a6-96ba-7c5226064e85" (UID: "a1fc9f6f-e597-41a6-96ba-7c5226064e85"). InnerVolumeSpecName "kube-api-access-ls8x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:37:27.237169 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.237118 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a1fc9f6f-e597-41a6-96ba-7c5226064e85" (UID: "a1fc9f6f-e597-41a6-96ba-7c5226064e85"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:37:27.335710 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.335663 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls8x6\" (UniqueName: \"kubernetes.io/projected/a1fc9f6f-e597-41a6-96ba-7c5226064e85-kube-api-access-ls8x6\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:27.335710 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.335705 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-trusted-ca-bundle\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:27.335710 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.335718 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-config\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:27.335918 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.335730 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-oauth-config\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:27.335918 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.335743 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1fc9f6f-e597-41a6-96ba-7c5226064e85-oauth-serving-cert\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:27.335918 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.335755 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1fc9f6f-e597-41a6-96ba-7c5226064e85-console-serving-cert\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:27.947345 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.947310 2573 generic.go:356] "Generic (PLEG): container finished" podID="a1fc9f6f-e597-41a6-96ba-7c5226064e85" containerID="9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8" exitCode=0 Jul 16 15:37:27.947345 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.947348 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdb6b68c4-m7dcm" Jul 16 15:37:27.947801 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.947364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdb6b68c4-m7dcm" event={"ID":"a1fc9f6f-e597-41a6-96ba-7c5226064e85","Type":"ContainerDied","Data":"9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8"} Jul 16 15:37:27.947801 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.947411 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdb6b68c4-m7dcm" event={"ID":"a1fc9f6f-e597-41a6-96ba-7c5226064e85","Type":"ContainerDied","Data":"42a8c9c79602d8ab046be1a7cb69c87cd6b5ef8d3ec6cde64eb99c3690348cc7"} Jul 16 15:37:27.947801 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.947444 2573 scope.go:117] "RemoveContainer" containerID="9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8" Jul 16 15:37:27.956379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.956362 2573 scope.go:117] "RemoveContainer" containerID="9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8" Jul 16 15:37:27.956666 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:37:27.956608 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8\": container with ID starting with 9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8 not found: ID does not exist" containerID="9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8" Jul 16 15:37:27.956718 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.956677 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8"} err="failed to get container status \"9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8\": rpc error: code = NotFound desc = could not find container \"9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8\": container with ID starting with 9fa7b17ed0be2162f7bfc62f55de7e894ddffdddbc192b6d9d6cc26a265ccaa8 not found: ID does not exist" Jul 16 15:37:27.968730 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.968712 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdb6b68c4-m7dcm"] Jul 16 15:37:27.972535 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:27.972514 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cdb6b68c4-m7dcm"] Jul 16 15:37:28.382824 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:28.382788 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1fc9f6f-e597-41a6-96ba-7c5226064e85" path="/var/lib/kubelet/pods/a1fc9f6f-e597-41a6-96ba-7c5226064e85/volumes" Jul 16 15:37:29.992688 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:29.992654 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:37:29.996380 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:29.996361 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8468d64cff-92r2h" Jul 16 15:37:34.847960 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:34.847924 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8k54x" Jul 16 15:37:40.601019 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.600978 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74d7c749d-xjlfg" podUID="da234646-0f6e-465e-b288-0e7c5a9ef002" containerName="console" containerID="cri-o://3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d" gracePeriod=15 Jul 16 15:37:40.848052 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.848030 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:37:40.956789 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.956702 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-service-ca\") pod \"da234646-0f6e-465e-b288-0e7c5a9ef002\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " Jul 16 15:37:40.956789 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.956777 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-oauth-config\") pod \"da234646-0f6e-465e-b288-0e7c5a9ef002\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " Jul 16 15:37:40.957005 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.956932 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-trusted-ca-bundle\") pod \"da234646-0f6e-465e-b288-0e7c5a9ef002\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " Jul 16 15:37:40.957005 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.956989 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-oauth-serving-cert\") pod \"da234646-0f6e-465e-b288-0e7c5a9ef002\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " Jul 16 15:37:40.957114 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957029 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-console-config\") pod \"da234646-0f6e-465e-b288-0e7c5a9ef002\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " Jul 16 15:37:40.957114 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957051 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-serving-cert\") pod \"da234646-0f6e-465e-b288-0e7c5a9ef002\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " Jul 16 15:37:40.957114 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8qtd\" (UniqueName: \"kubernetes.io/projected/da234646-0f6e-465e-b288-0e7c5a9ef002-kube-api-access-h8qtd\") pod \"da234646-0f6e-465e-b288-0e7c5a9ef002\" (UID: \"da234646-0f6e-465e-b288-0e7c5a9ef002\") " Jul 16 15:37:40.957114 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957084 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-service-ca" (OuterVolumeSpecName: "service-ca") pod "da234646-0f6e-465e-b288-0e7c5a9ef002" (UID: "da234646-0f6e-465e-b288-0e7c5a9ef002"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:37:40.957307 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957294 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "da234646-0f6e-465e-b288-0e7c5a9ef002" (UID: "da234646-0f6e-465e-b288-0e7c5a9ef002"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:37:40.957444 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957386 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da234646-0f6e-465e-b288-0e7c5a9ef002" (UID: "da234646-0f6e-465e-b288-0e7c5a9ef002"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:37:40.957444 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957399 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:40.957554 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957448 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-trusted-ca-bundle\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:40.957554 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.957451 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-console-config" (OuterVolumeSpecName: "console-config") pod "da234646-0f6e-465e-b288-0e7c5a9ef002" (UID: "da234646-0f6e-465e-b288-0e7c5a9ef002"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:37:40.959017 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.958992 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da234646-0f6e-465e-b288-0e7c5a9ef002" (UID: "da234646-0f6e-465e-b288-0e7c5a9ef002"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:37:40.959180 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.959159 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da234646-0f6e-465e-b288-0e7c5a9ef002" (UID: "da234646-0f6e-465e-b288-0e7c5a9ef002"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:37:40.959243 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.959187 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da234646-0f6e-465e-b288-0e7c5a9ef002-kube-api-access-h8qtd" (OuterVolumeSpecName: "kube-api-access-h8qtd") pod "da234646-0f6e-465e-b288-0e7c5a9ef002" (UID: "da234646-0f6e-465e-b288-0e7c5a9ef002"). InnerVolumeSpecName "kube-api-access-h8qtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:37:41.000010 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:40.999977 2573 generic.go:356] "Generic (PLEG): container finished" podID="da234646-0f6e-465e-b288-0e7c5a9ef002" containerID="3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d" exitCode=0 Jul 16 15:37:41.000181 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.000026 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d7c749d-xjlfg" Jul 16 15:37:41.000181 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.000036 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d7c749d-xjlfg" event={"ID":"da234646-0f6e-465e-b288-0e7c5a9ef002","Type":"ContainerDied","Data":"3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d"} Jul 16 15:37:41.000181 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.000086 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d7c749d-xjlfg" event={"ID":"da234646-0f6e-465e-b288-0e7c5a9ef002","Type":"ContainerDied","Data":"28a8e2119597b3982ccd75eafb4e7fa3d4cddc80d54976aa91e7f4c91bf28554"} Jul 16 15:37:41.000181 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.000113 2573 scope.go:117] "RemoveContainer" containerID="3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d" Jul 16 15:37:41.008823 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.008806 2573 scope.go:117] "RemoveContainer" containerID="3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d" Jul 16 15:37:41.009054 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:37:41.009037 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d\": container with ID starting with 3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d not found: ID does not exist" containerID="3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d" Jul 16 15:37:41.009110 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.009060 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d"} err="failed to get container status \"3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d\": rpc error: code = NotFound desc = could not find container \"3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d\": container with ID starting with 3f8f1862c6bb85a6f1090eb82f2b8d8cffe5d1acb78180bda2934bc6552a596d not found: ID does not exist" Jul 16 15:37:41.023648 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.023605 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d7c749d-xjlfg"] Jul 16 15:37:41.027585 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.027568 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74d7c749d-xjlfg"] Jul 16 15:37:41.058141 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.058121 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-oauth-config\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:41.058141 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.058139 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-oauth-serving-cert\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:41.058249 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.058150 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da234646-0f6e-465e-b288-0e7c5a9ef002-console-config\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:41.058249 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.058159 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da234646-0f6e-465e-b288-0e7c5a9ef002-console-serving-cert\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:41.058249 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:41.058166 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8qtd\" (UniqueName: \"kubernetes.io/projected/da234646-0f6e-465e-b288-0e7c5a9ef002-kube-api-access-h8qtd\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:37:42.382762 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:42.382736 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da234646-0f6e-465e-b288-0e7c5a9ef002" path="/var/lib/kubelet/pods/da234646-0f6e-465e-b288-0e7c5a9ef002/volumes" Jul 16 15:37:52.495905 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:52.495861 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:37:52.515968 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:52.515940 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:37:53.066872 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:37:53.066837 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:05.942564 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:05.942524 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:38:05.943047 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:05.942866 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="alertmanager" containerID="cri-o://89af2e4755b5c4332c60d6d3914bfba4a9a378d07985a488e2dffc69e0ae55f8" gracePeriod=120 Jul 16 15:38:05.943047 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:05.942936 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy-web" containerID="cri-o://9b1fca385218b91c608eed554b787d03edd99e06dd6d0680305ec28f951989e2" gracePeriod=120 Jul 16 15:38:05.943047 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:05.942930 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy-metric" containerID="cri-o://80c5c3b51fd8a21a1941ae92e3d0898d35cb72dcdcc9388e4c91aaba22a47fe7" gracePeriod=120 Jul 16 15:38:05.943047 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:05.942965 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy" containerID="cri-o://78da22e57eb0eaaef7b9f018465536b9659b5b9e13f36210802e1489f96778ae" gracePeriod=120 Jul 16 15:38:05.943047 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:05.942981 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="config-reloader" containerID="cri-o://5ef55611ab8a89f58a5e76a006cc7c8721bd26c36ae47b781107d5007fdd9f48" gracePeriod=120 Jul 16 15:38:05.943047 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:05.943008 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="prom-label-proxy" containerID="cri-o://05b5fdb006118d1f0fc373119ef8273281a5ca880c54772e86349bb2eb0d99d5" gracePeriod=120 Jul 16 15:38:06.105936 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:06.105910 2573 generic.go:356] "Generic (PLEG): container finished" podID="84211696-2c80-476e-9528-c09a71aa82de" containerID="05b5fdb006118d1f0fc373119ef8273281a5ca880c54772e86349bb2eb0d99d5" exitCode=0 Jul 16 15:38:06.105936 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:06.105931 2573 generic.go:356] "Generic (PLEG): container finished" podID="84211696-2c80-476e-9528-c09a71aa82de" containerID="78da22e57eb0eaaef7b9f018465536b9659b5b9e13f36210802e1489f96778ae" exitCode=0 Jul 16 15:38:06.105936 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:06.105937 2573 generic.go:356] "Generic (PLEG): container finished" podID="84211696-2c80-476e-9528-c09a71aa82de" containerID="5ef55611ab8a89f58a5e76a006cc7c8721bd26c36ae47b781107d5007fdd9f48" exitCode=0 Jul 16 15:38:06.106168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:06.105943 2573 generic.go:356] "Generic (PLEG): container finished" podID="84211696-2c80-476e-9528-c09a71aa82de" containerID="89af2e4755b5c4332c60d6d3914bfba4a9a378d07985a488e2dffc69e0ae55f8" exitCode=0 Jul 16 15:38:06.106168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:06.105983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerDied","Data":"05b5fdb006118d1f0fc373119ef8273281a5ca880c54772e86349bb2eb0d99d5"} Jul 16 15:38:06.106168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:06.106061 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerDied","Data":"78da22e57eb0eaaef7b9f018465536b9659b5b9e13f36210802e1489f96778ae"} Jul 16 15:38:06.106168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:06.106106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerDied","Data":"5ef55611ab8a89f58a5e76a006cc7c8721bd26c36ae47b781107d5007fdd9f48"} Jul 16 15:38:06.106168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:06.106148 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerDied","Data":"89af2e4755b5c4332c60d6d3914bfba4a9a378d07985a488e2dffc69e0ae55f8"} Jul 16 15:38:07.114082 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.114055 2573 generic.go:356] "Generic (PLEG): container finished" podID="84211696-2c80-476e-9528-c09a71aa82de" containerID="80c5c3b51fd8a21a1941ae92e3d0898d35cb72dcdcc9388e4c91aaba22a47fe7" exitCode=0 Jul 16 15:38:07.114082 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.114080 2573 generic.go:356] "Generic (PLEG): container finished" podID="84211696-2c80-476e-9528-c09a71aa82de" containerID="9b1fca385218b91c608eed554b787d03edd99e06dd6d0680305ec28f951989e2" exitCode=0 Jul 16 15:38:07.114409 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.114127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerDied","Data":"80c5c3b51fd8a21a1941ae92e3d0898d35cb72dcdcc9388e4c91aaba22a47fe7"} Jul 16 15:38:07.114409 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.114182 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerDied","Data":"9b1fca385218b91c608eed554b787d03edd99e06dd6d0680305ec28f951989e2"} Jul 16 15:38:07.189224 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.189203 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:07.286562 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.286487 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-metrics-client-ca\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.286562 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.286540 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-web\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.286757 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.286571 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.286757 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.286587 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-trusted-ca-bundle\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.286757 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.286628 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-cluster-tls-config\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.286757 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.286646 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-tls-assets\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.286757 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.286676 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-main-tls\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.286757 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.286707 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-config-out\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.287858 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.287399 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:07.287858 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.287493 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-web-config\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.289159 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.287531 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-config-volume\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.289159 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.288341 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvjnf\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-kube-api-access-jvjnf\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.289159 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.288386 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-main-db\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.289159 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.288398 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:07.289159 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.288421 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"84211696-2c80-476e-9528-c09a71aa82de\" (UID: \"84211696-2c80-476e-9528-c09a71aa82de\") " Jul 16 15:38:07.289159 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.289055 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 15:38:07.289887 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.289705 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:38:07.289986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.289919 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:07.290225 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.290167 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.290225 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.290195 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84211696-2c80-476e-9528-c09a71aa82de-metrics-client-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.291752 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.291348 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-config-out" (OuterVolumeSpecName: "config-out") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 15:38:07.291752 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.291591 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:07.297431 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.296560 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-kube-api-access-jvjnf" (OuterVolumeSpecName: "kube-api-access-jvjnf") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "kube-api-access-jvjnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:38:07.297431 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.297391 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:07.298164 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.298057 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:07.298164 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.298119 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-config-volume" (OuterVolumeSpecName: "config-volume") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:07.299940 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.299919 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:07.308433 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.308410 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-web-config" (OuterVolumeSpecName: "web-config") pod "84211696-2c80-476e-9528-c09a71aa82de" (UID: "84211696-2c80-476e-9528-c09a71aa82de"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:07.390623 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390597 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-config-out\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390689 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390634 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-web-config\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390689 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390643 2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-config-volume\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390689 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390652 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjnf\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-kube-api-access-jvjnf\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390689 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390661 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84211696-2c80-476e-9528-c09a71aa82de-alertmanager-main-db\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390689 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390671 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390689 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390681 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390689 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390690 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390898 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390699 2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-cluster-tls-config\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390898 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390707 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84211696-2c80-476e-9528-c09a71aa82de-tls-assets\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:07.390898 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:07.390715 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84211696-2c80-476e-9528-c09a71aa82de-secret-alertmanager-main-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:08.120553 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.120519 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84211696-2c80-476e-9528-c09a71aa82de","Type":"ContainerDied","Data":"b2aee9dba9676ef8e19e14de29cf5ae1a1884ce99fe9f02d1c3956bc58a82036"} Jul 16 15:38:08.120956 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.120597 2573 scope.go:117] "RemoveContainer" containerID="05b5fdb006118d1f0fc373119ef8273281a5ca880c54772e86349bb2eb0d99d5" Jul 16 15:38:08.120956 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.120529 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.128022 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.127925 2573 scope.go:117] "RemoveContainer" containerID="80c5c3b51fd8a21a1941ae92e3d0898d35cb72dcdcc9388e4c91aaba22a47fe7" Jul 16 15:38:08.136867 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.136793 2573 scope.go:117] "RemoveContainer" containerID="78da22e57eb0eaaef7b9f018465536b9659b5b9e13f36210802e1489f96778ae" Jul 16 15:38:08.143950 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.143927 2573 scope.go:117] "RemoveContainer" containerID="9b1fca385218b91c608eed554b787d03edd99e06dd6d0680305ec28f951989e2" Jul 16 15:38:08.150330 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.150312 2573 scope.go:117] "RemoveContainer" containerID="5ef55611ab8a89f58a5e76a006cc7c8721bd26c36ae47b781107d5007fdd9f48" Jul 16 15:38:08.156929 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.156915 2573 scope.go:117] "RemoveContainer" containerID="89af2e4755b5c4332c60d6d3914bfba4a9a378d07985a488e2dffc69e0ae55f8" Jul 16 15:38:08.161680 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.161659 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:38:08.164396 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.164380 2573 scope.go:117] "RemoveContainer" containerID="10f11609901add420b7aa96bad939480cec697c6a9e8d394b7cbc0309f342731" Jul 16 15:38:08.177340 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.177320 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:38:08.230237 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230212 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:38:08.230505 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230494 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="init-config-reloader" Jul 16 15:38:08.230551 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230507 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="init-config-reloader" Jul 16 15:38:08.230551 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230514 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy-web" Jul 16 15:38:08.230551 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230520 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy-web" Jul 16 15:38:08.230551 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230536 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy" Jul 16 15:38:08.230551 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230543 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy" Jul 16 15:38:08.230551 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230550 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1fc9f6f-e597-41a6-96ba-7c5226064e85" containerName="console" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230556 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1fc9f6f-e597-41a6-96ba-7c5226064e85" containerName="console" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230563 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="alertmanager" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230569 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="alertmanager" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230580 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da234646-0f6e-465e-b288-0e7c5a9ef002" containerName="console" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230585 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="da234646-0f6e-465e-b288-0e7c5a9ef002" containerName="console" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230590 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy-metric" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230595 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy-metric" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230601 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="prom-label-proxy" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230606 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="prom-label-proxy" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230636 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="config-reloader" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230643 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="config-reloader" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230689 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="prom-label-proxy" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230696 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="da234646-0f6e-465e-b288-0e7c5a9ef002" containerName="console" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230703 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy-metric" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230709 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230716 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="kube-rbac-proxy-web" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230721 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1fc9f6f-e597-41a6-96ba-7c5226064e85" containerName="console" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230729 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="alertmanager" Jul 16 15:38:08.230748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.230734 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84211696-2c80-476e-9528-c09a71aa82de" containerName="config-reloader" Jul 16 15:38:08.234572 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.234557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.240316 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.240300 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 15:38:08.242118 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.242098 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 15:38:08.242217 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.242098 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mvzws\"" Jul 16 15:38:08.245129 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.245116 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 15:38:08.245187 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.245144 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 15:38:08.245437 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.245422 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 15:38:08.245735 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.245714 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 15:38:08.245808 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.245794 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 15:38:08.246469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.246453 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 15:38:08.250293 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.250274 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 15:38:08.272344 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.272287 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:38:08.297974 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.297938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298077 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.297986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298077 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298146 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-config-out\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298146 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298146 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298233 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298233 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298292 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298292 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdps9\" (UniqueName: \"kubernetes.io/projected/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-kube-api-access-mdps9\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298353 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298385 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-web-config\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.298417 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.298383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.382717 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.382684 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84211696-2c80-476e-9528-c09a71aa82de" path="/var/lib/kubelet/pods/84211696-2c80-476e-9528-c09a71aa82de/volumes" Jul 16 15:38:08.399315 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399458 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399458 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdps9\" (UniqueName: \"kubernetes.io/projected/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-kube-api-access-mdps9\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399458 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399458 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-web-config\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399695 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399695 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399695 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399861 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399861 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-config-out\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.399861 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.400012 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.400012 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.400012 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.399904 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.400779 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.400429 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.400779 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.400583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.402383 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.402355 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.402493 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.402473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.402573 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.402548 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.403051 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.403020 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-web-config\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.403332 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.403312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.403407 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.403381 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.403476 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.403457 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.403853 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.403833 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.404252 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.404236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-config-out\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.417007 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.416981 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdps9\" (UniqueName: \"kubernetes.io/projected/5cd44bf4-cc3c-461b-ab91-825f7adffdaa-kube-api-access-mdps9\") pod \"alertmanager-main-0\" (UID: \"5cd44bf4-cc3c-461b-ab91-825f7adffdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.543746 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.543689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 15:38:08.711359 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:08.711324 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 15:38:08.714533 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:38:08.714496 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd44bf4_cc3c_461b_ab91_825f7adffdaa.slice/crio-47104b10e15e98dd5403a06b46112ab7aaa02db0f08c96f1db5dde8415ae3308 WatchSource:0}: Error finding container 47104b10e15e98dd5403a06b46112ab7aaa02db0f08c96f1db5dde8415ae3308: Status 404 returned error can't find the container with id 47104b10e15e98dd5403a06b46112ab7aaa02db0f08c96f1db5dde8415ae3308 Jul 16 15:38:09.126330 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:09.126257 2573 generic.go:356] "Generic (PLEG): container finished" podID="5cd44bf4-cc3c-461b-ab91-825f7adffdaa" containerID="fb54a20f37171bcd50d279656df3a8c57ddf10223f052644835b2f3cf0611ccb" exitCode=0 Jul 16 15:38:09.126719 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:09.126347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5cd44bf4-cc3c-461b-ab91-825f7adffdaa","Type":"ContainerDied","Data":"fb54a20f37171bcd50d279656df3a8c57ddf10223f052644835b2f3cf0611ccb"} Jul 16 15:38:09.126719 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:09.126441 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5cd44bf4-cc3c-461b-ab91-825f7adffdaa","Type":"ContainerStarted","Data":"47104b10e15e98dd5403a06b46112ab7aaa02db0f08c96f1db5dde8415ae3308"} Jul 16 15:38:10.135384 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.135350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5cd44bf4-cc3c-461b-ab91-825f7adffdaa","Type":"ContainerStarted","Data":"53578d509fd8c7bc37e700e73200b474a4e0712ade12aceaf6c95562127f996c"} Jul 16 15:38:10.135765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.135446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5cd44bf4-cc3c-461b-ab91-825f7adffdaa","Type":"ContainerStarted","Data":"df0bf22d426d28809b748701a40f43064110e62f38fe099ea994d07fbb823cf0"} Jul 16 15:38:10.135765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.135478 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5cd44bf4-cc3c-461b-ab91-825f7adffdaa","Type":"ContainerStarted","Data":"832387110b88d97c2689fe69cad3fa0ae0c66268514bcad98aeb662dc1ed324f"} Jul 16 15:38:10.135765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.135502 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5cd44bf4-cc3c-461b-ab91-825f7adffdaa","Type":"ContainerStarted","Data":"1549cdfbd877332751667aed38e36a1141cd0bb5c3d572e26eb98f8b5f21b3d6"} Jul 16 15:38:10.135765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.135524 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5cd44bf4-cc3c-461b-ab91-825f7adffdaa","Type":"ContainerStarted","Data":"65f2f6868cdfaed341ee2fa6618c14be9c57b40863da7d46c62003da0fadd497"} Jul 16 15:38:10.135765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.135547 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5cd44bf4-cc3c-461b-ab91-825f7adffdaa","Type":"ContainerStarted","Data":"bcaac31aa81148e9c557057858235c58bf752d556745b5c1bc2a9982e068cd32"} Jul 16 15:38:10.172035 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.171972 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.157583145 podStartE2EDuration="2.171952616s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.369471ms" isStatefulPod=false podCreationTimestamp="2026-07-16 15:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:38:10.171750531 +0000 UTC m=+144.610277355" watchObservedRunningTime="2026-07-16 15:38:10.171952616 +0000 UTC m=+144.610479439" Jul 16 15:38:10.232858 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.232396 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:38:10.232858 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.232778 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="prometheus" containerID="cri-o://341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" gracePeriod=600 Jul 16 15:38:10.233108 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.232911 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="thanos-sidecar" containerID="cri-o://8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" gracePeriod=600 Jul 16 15:38:10.233108 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.232943 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="config-reloader" containerID="cri-o://6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" gracePeriod=600 Jul 16 15:38:10.233108 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.233025 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy-web" containerID="cri-o://54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" gracePeriod=600 Jul 16 15:38:10.233108 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.233018 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy" containerID="cri-o://05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" gracePeriod=600 Jul 16 15:38:10.233108 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.233060 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" gracePeriod=600 Jul 16 15:38:10.489443 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.489417 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:10.619102 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619075 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-db\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619102 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619107 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-metrics-client-ca\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619145 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619168 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-2\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619193 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config-out\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619219 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-kubelet-serving-ca-bundle\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619251 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619290 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619326 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-1\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619379 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-grpc-tls\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619790 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619401 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-tls\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619790 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619440 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chw5g\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-kube-api-access-chw5g\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619790 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619479 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-web-config\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.619790 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619474 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:10.619790 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619497 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:10.619790 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619525 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-thanos-prometheus-http-client-file\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.620080 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.619787 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:10.620183 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.620153 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:10.620392 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.620365 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.620583 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-kube-rbac-proxy\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.620653 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-trusted-ca-bundle\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.620692 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-metrics-client-certs\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.620725 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-tls-assets\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.620758 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-serving-certs-ca-bundle\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.620795 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-0\") pod \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\" (UID: \"c6daed1a-9ca4-4f1f-9714-b27afc543ab3\") " Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.621080 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.621098 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-db\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.621115 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-metrics-client-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.621130 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.621880 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.621145 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.622750 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.622718 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:10.623019 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.622981 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:10.623319 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.623289 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config-out" (OuterVolumeSpecName: "config-out") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 15:38:10.623506 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.623469 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.623756 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.623718 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.623756 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.623748 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:38:10.623916 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.623800 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config" (OuterVolumeSpecName: "config") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.623969 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.623917 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.624306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.624279 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-kube-api-access-chw5g" (OuterVolumeSpecName: "kube-api-access-chw5g") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "kube-api-access-chw5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:38:10.624378 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.624354 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:38:10.624549 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.624528 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.624647 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.624604 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.625111 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.625086 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.625335 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.625315 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.634504 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.634485 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-web-config" (OuterVolumeSpecName: "web-config") pod "c6daed1a-9ca4-4f1f-9714-b27afc543ab3" (UID: "c6daed1a-9ca4-4f1f-9714-b27afc543ab3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:38:10.722060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.721998 2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-grpc-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722022 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722032 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chw5g\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-kube-api-access-chw5g\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722041 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-web-config\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722050 2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722060 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722060 2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-kube-rbac-proxy\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722068 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722078 2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-metrics-client-certs\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722088 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-tls-assets\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722096 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722104 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722112 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722121 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config-out\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722128 2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-config\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:10.722302 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:10.722136 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6daed1a-9ca4-4f1f-9714-b27afc543ab3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:38:11.142888 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142856 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" exitCode=0 Jul 16 15:38:11.142888 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142879 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" exitCode=0 Jul 16 15:38:11.142888 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142885 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" exitCode=0 Jul 16 15:38:11.142888 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142891 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" exitCode=0 Jul 16 15:38:11.142888 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142896 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" exitCode=0 Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142902 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" exitCode=0 Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142930 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerDied","Data":"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964"} Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142949 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.142992 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerDied","Data":"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32"} Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.143020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerDied","Data":"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67"} Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.143044 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerDied","Data":"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7"} Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.143062 2573 scope.go:117] "RemoveContainer" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.143069 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerDied","Data":"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c"} Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.143187 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerDied","Data":"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8"} Jul 16 15:38:11.143475 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.143233 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6daed1a-9ca4-4f1f-9714-b27afc543ab3","Type":"ContainerDied","Data":"604247fc120b47a6ae94ce325d813e5e7a502c373b2e86397b2cfc590e7aff22"} Jul 16 15:38:11.150684 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.150664 2573 scope.go:117] "RemoveContainer" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" Jul 16 15:38:11.157284 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.157267 2573 scope.go:117] "RemoveContainer" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" Jul 16 15:38:11.163805 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.163790 2573 scope.go:117] "RemoveContainer" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" Jul 16 15:38:11.170812 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.170787 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:38:11.171418 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.171391 2573 scope.go:117] "RemoveContainer" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" Jul 16 15:38:11.175579 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.175560 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:38:11.178699 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.178682 2573 scope.go:117] "RemoveContainer" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" Jul 16 15:38:11.185442 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.185428 2573 scope.go:117] "RemoveContainer" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" Jul 16 15:38:11.191604 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.191589 2573 scope.go:117] "RemoveContainer" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" Jul 16 15:38:11.191849 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:38:11.191832 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": container with ID starting with da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964 not found: ID does not exist" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" Jul 16 15:38:11.191888 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.191856 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964"} err="failed to get container status \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": rpc error: code = NotFound desc = could not find container \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": container with ID starting with da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964 not found: ID does not exist" Jul 16 15:38:11.191888 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.191879 2573 scope.go:117] "RemoveContainer" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" Jul 16 15:38:11.192104 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:38:11.192086 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": container with ID starting with 05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32 not found: ID does not exist" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" Jul 16 15:38:11.192154 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.192110 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32"} err="failed to get container status \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": rpc error: code = NotFound desc = could not find container \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": container with ID starting with 05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32 not found: ID does not exist" Jul 16 15:38:11.192154 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.192139 2573 scope.go:117] "RemoveContainer" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" Jul 16 15:38:11.192354 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:38:11.192337 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": container with ID starting with 54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67 not found: ID does not exist" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" Jul 16 15:38:11.192389 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.192359 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67"} err="failed to get container status \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": rpc error: code = NotFound desc = could not find container \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": container with ID starting with 54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67 not found: ID does not exist" Jul 16 15:38:11.192389 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.192373 2573 scope.go:117] "RemoveContainer" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" Jul 16 15:38:11.192681 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:38:11.192606 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": container with ID starting with 8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7 not found: ID does not exist" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" Jul 16 15:38:11.192720 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.192691 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7"} err="failed to get container status \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": rpc error: code = NotFound desc = could not find container \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": container with ID starting with 8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7 not found: ID does not exist" Jul 16 15:38:11.192760 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.192712 2573 scope.go:117] "RemoveContainer" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" Jul 16 15:38:11.192963 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:38:11.192945 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": container with ID starting with 6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c not found: ID does not exist" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" Jul 16 15:38:11.193004 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.192967 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c"} err="failed to get container status \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": rpc error: code = NotFound desc = could not find container \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": container with ID starting with 6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c not found: ID does not exist" Jul 16 15:38:11.193004 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.192980 2573 scope.go:117] "RemoveContainer" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" Jul 16 15:38:11.193193 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:38:11.193178 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": container with ID starting with 341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8 not found: ID does not exist" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" Jul 16 15:38:11.193228 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.193196 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8"} err="failed to get container status \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": rpc error: code = NotFound desc = could not find container \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": container with ID starting with 341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8 not found: ID does not exist" Jul 16 15:38:11.193228 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.193208 2573 scope.go:117] "RemoveContainer" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" Jul 16 15:38:11.193418 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:38:11.193403 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": container with ID starting with ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367 not found: ID does not exist" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" Jul 16 15:38:11.193453 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.193421 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367"} err="failed to get container status \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": rpc error: code = NotFound desc = could not find container \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": container with ID starting with ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367 not found: ID does not exist" Jul 16 15:38:11.193453 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.193434 2573 scope.go:117] "RemoveContainer" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" Jul 16 15:38:11.193653 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.193635 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964"} err="failed to get container status \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": rpc error: code = NotFound desc = could not find container \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": container with ID starting with da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964 not found: ID does not exist" Jul 16 15:38:11.193709 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.193654 2573 scope.go:117] "RemoveContainer" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" Jul 16 15:38:11.193884 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.193865 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32"} err="failed to get container status \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": rpc error: code = NotFound desc = could not find container \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": container with ID starting with 05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32 not found: ID does not exist" Jul 16 15:38:11.193984 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.193884 2573 scope.go:117] "RemoveContainer" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" Jul 16 15:38:11.194089 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194072 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67"} err="failed to get container status \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": rpc error: code = NotFound desc = could not find container \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": container with ID starting with 54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67 not found: ID does not exist" Jul 16 15:38:11.194124 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194089 2573 scope.go:117] "RemoveContainer" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" Jul 16 15:38:11.194312 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194294 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7"} err="failed to get container status \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": rpc error: code = NotFound desc = could not find container \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": container with ID starting with 8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7 not found: ID does not exist" Jul 16 15:38:11.194353 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194314 2573 scope.go:117] "RemoveContainer" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" Jul 16 15:38:11.194507 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194490 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c"} err="failed to get container status \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": rpc error: code = NotFound desc = could not find container \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": container with ID starting with 6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c not found: ID does not exist" Jul 16 15:38:11.194548 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194507 2573 scope.go:117] "RemoveContainer" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" Jul 16 15:38:11.194724 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194705 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8"} err="failed to get container status \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": rpc error: code = NotFound desc = could not find container \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": container with ID starting with 341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8 not found: ID does not exist" Jul 16 15:38:11.194762 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194725 2573 scope.go:117] "RemoveContainer" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" Jul 16 15:38:11.194924 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194908 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367"} err="failed to get container status \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": rpc error: code = NotFound desc = could not find container \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": container with ID starting with ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367 not found: ID does not exist" Jul 16 15:38:11.194924 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.194922 2573 scope.go:117] "RemoveContainer" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" Jul 16 15:38:11.195113 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195086 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964"} err="failed to get container status \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": rpc error: code = NotFound desc = could not find container \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": container with ID starting with da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964 not found: ID does not exist" Jul 16 15:38:11.195113 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195113 2573 scope.go:117] "RemoveContainer" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" Jul 16 15:38:11.195321 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195306 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32"} err="failed to get container status \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": rpc error: code = NotFound desc = could not find container \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": container with ID starting with 05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32 not found: ID does not exist" Jul 16 15:38:11.195365 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195321 2573 scope.go:117] "RemoveContainer" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" Jul 16 15:38:11.195534 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195516 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67"} err="failed to get container status \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": rpc error: code = NotFound desc = could not find container \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": container with ID starting with 54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67 not found: ID does not exist" Jul 16 15:38:11.195606 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195536 2573 scope.go:117] "RemoveContainer" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" Jul 16 15:38:11.195758 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195738 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7"} err="failed to get container status \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": rpc error: code = NotFound desc = could not find container \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": container with ID starting with 8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7 not found: ID does not exist" Jul 16 15:38:11.195812 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195759 2573 scope.go:117] "RemoveContainer" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" Jul 16 15:38:11.195940 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195922 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c"} err="failed to get container status \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": rpc error: code = NotFound desc = could not find container \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": container with ID starting with 6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c not found: ID does not exist" Jul 16 15:38:11.195997 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.195942 2573 scope.go:117] "RemoveContainer" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" Jul 16 15:38:11.196181 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.196166 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8"} err="failed to get container status \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": rpc error: code = NotFound desc = could not find container \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": container with ID starting with 341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8 not found: ID does not exist" Jul 16 15:38:11.196250 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.196183 2573 scope.go:117] "RemoveContainer" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" Jul 16 15:38:11.196380 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.196364 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367"} err="failed to get container status \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": rpc error: code = NotFound desc = could not find container \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": container with ID starting with ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367 not found: ID does not exist" Jul 16 15:38:11.196443 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.196381 2573 scope.go:117] "RemoveContainer" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" Jul 16 15:38:11.196580 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.196565 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964"} err="failed to get container status \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": rpc error: code = NotFound desc = could not find container \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": container with ID starting with da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964 not found: ID does not exist" Jul 16 15:38:11.196658 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.196582 2573 scope.go:117] "RemoveContainer" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" Jul 16 15:38:11.196857 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.196839 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32"} err="failed to get container status \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": rpc error: code = NotFound desc = could not find container \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": container with ID starting with 05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32 not found: ID does not exist" Jul 16 15:38:11.196925 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.196858 2573 scope.go:117] "RemoveContainer" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" Jul 16 15:38:11.197041 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197026 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67"} err="failed to get container status \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": rpc error: code = NotFound desc = could not find container \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": container with ID starting with 54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67 not found: ID does not exist" Jul 16 15:38:11.197092 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197040 2573 scope.go:117] "RemoveContainer" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" Jul 16 15:38:11.197238 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197215 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7"} err="failed to get container status \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": rpc error: code = NotFound desc = could not find container \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": container with ID starting with 8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7 not found: ID does not exist" Jul 16 15:38:11.197238 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197232 2573 scope.go:117] "RemoveContainer" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" Jul 16 15:38:11.197439 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197423 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c"} err="failed to get container status \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": rpc error: code = NotFound desc = could not find container \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": container with ID starting with 6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c not found: ID does not exist" Jul 16 15:38:11.197439 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197438 2573 scope.go:117] "RemoveContainer" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" Jul 16 15:38:11.197636 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197602 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8"} err="failed to get container status \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": rpc error: code = NotFound desc = could not find container \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": container with ID starting with 341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8 not found: ID does not exist" Jul 16 15:38:11.197636 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197634 2573 scope.go:117] "RemoveContainer" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" Jul 16 15:38:11.197832 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197816 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367"} err="failed to get container status \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": rpc error: code = NotFound desc = could not find container \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": container with ID starting with ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367 not found: ID does not exist" Jul 16 15:38:11.197832 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.197831 2573 scope.go:117] "RemoveContainer" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" Jul 16 15:38:11.198058 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198042 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964"} err="failed to get container status \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": rpc error: code = NotFound desc = could not find container \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": container with ID starting with da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964 not found: ID does not exist" Jul 16 15:38:11.198058 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198058 2573 scope.go:117] "RemoveContainer" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" Jul 16 15:38:11.198281 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198264 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32"} err="failed to get container status \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": rpc error: code = NotFound desc = could not find container \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": container with ID starting with 05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32 not found: ID does not exist" Jul 16 15:38:11.198281 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198280 2573 scope.go:117] "RemoveContainer" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" Jul 16 15:38:11.198458 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198443 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67"} err="failed to get container status \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": rpc error: code = NotFound desc = could not find container \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": container with ID starting with 54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67 not found: ID does not exist" Jul 16 15:38:11.198547 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198458 2573 scope.go:117] "RemoveContainer" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" Jul 16 15:38:11.198674 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198659 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7"} err="failed to get container status \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": rpc error: code = NotFound desc = could not find container \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": container with ID starting with 8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7 not found: ID does not exist" Jul 16 15:38:11.198722 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198675 2573 scope.go:117] "RemoveContainer" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" Jul 16 15:38:11.198862 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198847 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c"} err="failed to get container status \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": rpc error: code = NotFound desc = could not find container \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": container with ID starting with 6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c not found: ID does not exist" Jul 16 15:38:11.198903 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.198863 2573 scope.go:117] "RemoveContainer" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" Jul 16 15:38:11.199036 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199021 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8"} err="failed to get container status \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": rpc error: code = NotFound desc = could not find container \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": container with ID starting with 341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8 not found: ID does not exist" Jul 16 15:38:11.199077 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199036 2573 scope.go:117] "RemoveContainer" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" Jul 16 15:38:11.199177 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199163 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367"} err="failed to get container status \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": rpc error: code = NotFound desc = could not find container \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": container with ID starting with ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367 not found: ID does not exist" Jul 16 15:38:11.199220 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199177 2573 scope.go:117] "RemoveContainer" containerID="da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964" Jul 16 15:38:11.199384 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199369 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964"} err="failed to get container status \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": rpc error: code = NotFound desc = could not find container \"da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964\": container with ID starting with da46d62010c030337953ba840bbce5662736acefbb85229e4c62490e9d263964 not found: ID does not exist" Jul 16 15:38:11.199384 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199383 2573 scope.go:117] "RemoveContainer" containerID="05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32" Jul 16 15:38:11.199558 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199542 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32"} err="failed to get container status \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": rpc error: code = NotFound desc = could not find container \"05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32\": container with ID starting with 05f5b492a002e3404b6fbc3f03d157f6446922cc8d58361d41b881e586cb3c32 not found: ID does not exist" Jul 16 15:38:11.199599 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199559 2573 scope.go:117] "RemoveContainer" containerID="54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67" Jul 16 15:38:11.199741 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199726 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67"} err="failed to get container status \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": rpc error: code = NotFound desc = could not find container \"54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67\": container with ID starting with 54c821b66ba187de901cd4952a2de67bf13c75fd413732f38480f8f3c242ca67 not found: ID does not exist" Jul 16 15:38:11.199779 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199740 2573 scope.go:117] "RemoveContainer" containerID="8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7" Jul 16 15:38:11.199912 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199897 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7"} err="failed to get container status \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": rpc error: code = NotFound desc = could not find container \"8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7\": container with ID starting with 8c78a3868ff36b57b2352e5f93f339dd9ab0407175b1be411e1bd0baa8bcc0a7 not found: ID does not exist" Jul 16 15:38:11.199950 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.199912 2573 scope.go:117] "RemoveContainer" containerID="6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c" Jul 16 15:38:11.200077 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.200063 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c"} err="failed to get container status \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": rpc error: code = NotFound desc = could not find container \"6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c\": container with ID starting with 6235edc8f54600ac5473cd1aa7a9a59b976e244edda410d6b9afd334ebf86b5c not found: ID does not exist" Jul 16 15:38:11.200113 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.200077 2573 scope.go:117] "RemoveContainer" containerID="341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8" Jul 16 15:38:11.200242 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.200228 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8"} err="failed to get container status \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": rpc error: code = NotFound desc = could not find container \"341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8\": container with ID starting with 341b83afe53076a4b416041a168ee7cad193e86ea8ffa1b4e5bf3703045419e8 not found: ID does not exist" Jul 16 15:38:11.200277 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.200242 2573 scope.go:117] "RemoveContainer" containerID="ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367" Jul 16 15:38:11.200405 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.200391 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367"} err="failed to get container status \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": rpc error: code = NotFound desc = could not find container \"ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367\": container with ID starting with ba227a375893fe53f6f62298f956fa0e73de9985de42a4e967e2bf46f9dc2367 not found: ID does not exist" Jul 16 15:38:11.235091 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235064 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:38:11.235401 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235386 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="prometheus" Jul 16 15:38:11.235468 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235405 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="prometheus" Jul 16 15:38:11.235468 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235432 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="thanos-sidecar" Jul 16 15:38:11.235468 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235441 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="thanos-sidecar" Jul 16 15:38:11.235468 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235453 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy-web" Jul 16 15:38:11.235468 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235462 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy-web" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235480 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235488 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235499 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy-thanos" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235507 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy-thanos" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235519 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="config-reloader" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235527 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="config-reloader" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235538 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="init-config-reloader" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235546 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="init-config-reloader" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235632 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy-web" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235646 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="thanos-sidecar" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235657 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy-thanos" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235669 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="kube-rbac-proxy" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235678 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="prometheus" Jul 16 15:38:11.235737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.235688 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" containerName="config-reloader" Jul 16 15:38:11.243048 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.243029 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.248453 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.248435 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 15:38:11.248655 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.248629 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 15:38:11.248775 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.248759 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 15:38:11.249265 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.249107 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 15:38:11.249265 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.249208 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 15:38:11.249517 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.249474 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 15:38:11.249852 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.249835 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 15:38:11.250609 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.250591 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 15:38:11.252323 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.252288 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 15:38:11.252323 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.252310 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cvc984r5dp3hi\"" Jul 16 15:38:11.252501 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.252334 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 15:38:11.252501 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.252290 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 15:38:11.252501 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.252297 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5s8ks\"" Jul 16 15:38:11.252793 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.252777 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 15:38:11.253153 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.253138 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 15:38:11.260026 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.260006 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 15:38:11.262168 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.262150 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:38:11.326949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.326873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.326949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.326906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-web-config\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.326949 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.326927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.326969 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjh6\" (UniqueName: \"kubernetes.io/projected/b918f262-9202-4a35-8f85-169a7d68f627-kube-api-access-8mjh6\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b918f262-9202-4a35-8f85-169a7d68f627-config-out\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327127 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-config\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b918f262-9202-4a35-8f85-169a7d68f627-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327725 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327725 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.327725 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.327528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428511 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428511 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428751 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428751 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428751 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428751 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428751 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428751 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428751 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-web-config\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.428751 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjh6\" (UniqueName: \"kubernetes.io/projected/b918f262-9202-4a35-8f85-169a7d68f627-kube-api-access-8mjh6\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b918f262-9202-4a35-8f85-169a7d68f627-config-out\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.428959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-config\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429488 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b918f262-9202-4a35-8f85-169a7d68f627-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429488 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429488 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429231 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429488 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429410 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429684 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429591 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429684 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429643 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.429922 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.429896 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.430279 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.430253 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.432285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.432073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.432373 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.432357 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b918f262-9202-4a35-8f85-169a7d68f627-config-out\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.432501 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.432453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.432743 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.432719 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-web-config\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.432826 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.432793 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.433056 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.433031 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.433183 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.433152 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b918f262-9202-4a35-8f85-169a7d68f627-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.433539 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.433517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.434286 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.434264 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.434502 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.434488 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b918f262-9202-4a35-8f85-169a7d68f627-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.434919 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.434904 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-config\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.435281 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.435262 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b918f262-9202-4a35-8f85-169a7d68f627-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.445162 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.445143 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjh6\" (UniqueName: \"kubernetes.io/projected/b918f262-9202-4a35-8f85-169a7d68f627-kube-api-access-8mjh6\") pod \"prometheus-k8s-0\" (UID: \"b918f262-9202-4a35-8f85-169a7d68f627\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.552765 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.552737 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:38:11.690682 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:11.690577 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 15:38:11.692796 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:38:11.692769 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb918f262_9202_4a35_8f85_169a7d68f627.slice/crio-f667635295b7f3675fa73596fc7c06b842434170ae406e7f578aa672a950f33e WatchSource:0}: Error finding container f667635295b7f3675fa73596fc7c06b842434170ae406e7f578aa672a950f33e: Status 404 returned error can't find the container with id f667635295b7f3675fa73596fc7c06b842434170ae406e7f578aa672a950f33e Jul 16 15:38:12.150379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:12.150347 2573 generic.go:356] "Generic (PLEG): container finished" podID="b918f262-9202-4a35-8f85-169a7d68f627" containerID="3ccbe43e597f2a290d607025ac3b97a29c34baad441ccac8e56e6605afa73800" exitCode=0 Jul 16 15:38:12.150830 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:12.150424 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b918f262-9202-4a35-8f85-169a7d68f627","Type":"ContainerDied","Data":"3ccbe43e597f2a290d607025ac3b97a29c34baad441ccac8e56e6605afa73800"} Jul 16 15:38:12.150830 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:12.150484 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b918f262-9202-4a35-8f85-169a7d68f627","Type":"ContainerStarted","Data":"f667635295b7f3675fa73596fc7c06b842434170ae406e7f578aa672a950f33e"} Jul 16 15:38:12.386737 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:12.386603 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6daed1a-9ca4-4f1f-9714-b27afc543ab3" path="/var/lib/kubelet/pods/c6daed1a-9ca4-4f1f-9714-b27afc543ab3/volumes" Jul 16 15:38:13.162481 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:13.162444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b918f262-9202-4a35-8f85-169a7d68f627","Type":"ContainerStarted","Data":"fdf565d7aaa7fd84506d902c62840b581ac7857ffb93079c5a4bbb768327058b"} Jul 16 15:38:13.162852 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:13.162544 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b918f262-9202-4a35-8f85-169a7d68f627","Type":"ContainerStarted","Data":"dfa0fbd470c71c7d30cef7ed61110c164ab7073c3ee07ba74afc11e316e993cd"} Jul 16 15:38:13.162852 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:13.162574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b918f262-9202-4a35-8f85-169a7d68f627","Type":"ContainerStarted","Data":"b78e18e26e38cabc782199cffcbd11f75ca4a317431ed9e977d1088144cd4e5d"} Jul 16 15:38:13.162852 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:13.162604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b918f262-9202-4a35-8f85-169a7d68f627","Type":"ContainerStarted","Data":"9f79ce92aecd3b0a3beb79c9ae12810409b95e2dc325f2cf08f98f5e54924836"} Jul 16 15:38:13.162852 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:13.162665 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b918f262-9202-4a35-8f85-169a7d68f627","Type":"ContainerStarted","Data":"573ad3d16f5a94b516bfee74d82cbca29e7d4b2c1872902adceea02ca61c0542"} Jul 16 15:38:13.162852 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:13.162694 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b918f262-9202-4a35-8f85-169a7d68f627","Type":"ContainerStarted","Data":"ab422193aa3534036d2161e96906e22aea3665eec9d8cfe6ce0ba39456845881"} Jul 16 15:38:13.196529 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:13.196475 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.178165483 podStartE2EDuration="2.196410415s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.244932ms" isStatefulPod=false podCreationTimestamp="2026-07-16 15:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:38:13.193373297 +0000 UTC m=+147.631900121" watchObservedRunningTime="2026-07-16 15:38:13.196410415 +0000 UTC m=+147.634937237" Jul 16 15:38:16.552819 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:38:16.552788 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:39:11.553076 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:11.553041 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:39:11.568363 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:11.568335 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:39:12.427442 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:12.427411 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 15:39:27.968080 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:27.968047 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v6wnq"] Jul 16 15:39:27.972416 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:27.972397 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:27.979417 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:27.979397 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 15:39:27.981417 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:27.981399 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 15:39:27.986412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:27.986390 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v6wnq"] Jul 16 15:39:28.091625 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.091590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/543537e6-98cb-4613-ba51-c79f2dab703f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v6wnq\" (UID: \"543537e6-98cb-4613-ba51-c79f2dab703f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:28.091808 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.091747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhm6\" (UniqueName: \"kubernetes.io/projected/543537e6-98cb-4613-ba51-c79f2dab703f-kube-api-access-qbhm6\") pod \"cert-manager-webhook-6486ffd997-v6wnq\" (UID: \"543537e6-98cb-4613-ba51-c79f2dab703f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:28.192705 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.192668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/543537e6-98cb-4613-ba51-c79f2dab703f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v6wnq\" (UID: \"543537e6-98cb-4613-ba51-c79f2dab703f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:28.192875 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.192757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhm6\" (UniqueName: \"kubernetes.io/projected/543537e6-98cb-4613-ba51-c79f2dab703f-kube-api-access-qbhm6\") pod \"cert-manager-webhook-6486ffd997-v6wnq\" (UID: \"543537e6-98cb-4613-ba51-c79f2dab703f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:28.209037 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.209008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/543537e6-98cb-4613-ba51-c79f2dab703f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v6wnq\" (UID: \"543537e6-98cb-4613-ba51-c79f2dab703f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:28.209156 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.209039 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhm6\" (UniqueName: \"kubernetes.io/projected/543537e6-98cb-4613-ba51-c79f2dab703f-kube-api-access-qbhm6\") pod \"cert-manager-webhook-6486ffd997-v6wnq\" (UID: \"543537e6-98cb-4613-ba51-c79f2dab703f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:28.293430 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.293399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:28.422310 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.422287 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v6wnq"] Jul 16 15:39:28.424428 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:39:28.424409 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod543537e6_98cb_4613_ba51_c79f2dab703f.slice/crio-f05df3b895601059d254632098e11bdea83a1ae8b83194d12be884ccc9de7a08 WatchSource:0}: Error finding container f05df3b895601059d254632098e11bdea83a1ae8b83194d12be884ccc9de7a08: Status 404 returned error can't find the container with id f05df3b895601059d254632098e11bdea83a1ae8b83194d12be884ccc9de7a08 Jul 16 15:39:28.475296 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:28.475266 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" event={"ID":"543537e6-98cb-4613-ba51-c79f2dab703f","Type":"ContainerStarted","Data":"f05df3b895601059d254632098e11bdea83a1ae8b83194d12be884ccc9de7a08"} Jul 16 15:39:32.493631 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:32.493584 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" event={"ID":"543537e6-98cb-4613-ba51-c79f2dab703f","Type":"ContainerStarted","Data":"49a93d4a091499250d9b9923e3109393b45d18e022fb66d0b3ba55c9cc103174"} Jul 16 15:39:32.493995 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:32.493668 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:32.517882 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:32.517839 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" podStartSLOduration=2.4533785 podStartE2EDuration="5.517825869s" totalImagesPullingTime="3.064447369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:39:32.516918801 +0000 UTC m=+226.955445624" watchObservedRunningTime="2026-07-16 15:39:32.517825869 +0000 UTC m=+226.956352691" Jul 16 15:39:38.500092 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:38.500063 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v6wnq" Jul 16 15:39:41.311841 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.311807 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk"] Jul 16 15:39:41.314958 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.314943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" Jul 16 15:39:41.319858 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.319834 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vwxsc\"" Jul 16 15:39:41.319858 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.319849 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 15:39:41.320748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.320733 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 15:39:41.331236 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.331214 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk"] Jul 16 15:39:41.409969 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.409942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b29c0bf-6034-43e7-b04c-1dd7156bfe72-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ggnfk\" (UID: \"6b29c0bf-6034-43e7-b04c-1dd7156bfe72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" Jul 16 15:39:41.410080 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.409974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqzw\" (UniqueName: \"kubernetes.io/projected/6b29c0bf-6034-43e7-b04c-1dd7156bfe72-kube-api-access-szqzw\") pod \"openshift-lws-operator-fd8ccff4c-ggnfk\" (UID: \"6b29c0bf-6034-43e7-b04c-1dd7156bfe72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" Jul 16 15:39:41.511285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.511250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b29c0bf-6034-43e7-b04c-1dd7156bfe72-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ggnfk\" (UID: \"6b29c0bf-6034-43e7-b04c-1dd7156bfe72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" Jul 16 15:39:41.511285 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.511287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqzw\" (UniqueName: \"kubernetes.io/projected/6b29c0bf-6034-43e7-b04c-1dd7156bfe72-kube-api-access-szqzw\") pod \"openshift-lws-operator-fd8ccff4c-ggnfk\" (UID: \"6b29c0bf-6034-43e7-b04c-1dd7156bfe72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" Jul 16 15:39:41.511645 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.511603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b29c0bf-6034-43e7-b04c-1dd7156bfe72-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ggnfk\" (UID: \"6b29c0bf-6034-43e7-b04c-1dd7156bfe72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" Jul 16 15:39:41.526420 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.526391 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqzw\" (UniqueName: \"kubernetes.io/projected/6b29c0bf-6034-43e7-b04c-1dd7156bfe72-kube-api-access-szqzw\") pod \"openshift-lws-operator-fd8ccff4c-ggnfk\" (UID: \"6b29c0bf-6034-43e7-b04c-1dd7156bfe72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" Jul 16 15:39:41.624512 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.624432 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" Jul 16 15:39:41.776532 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:41.776500 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk"] Jul 16 15:39:41.779328 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:39:41.779300 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b29c0bf_6034_43e7_b04c_1dd7156bfe72.slice/crio-e25540948ab73fd7bb0680b091e12c36f49aae8fda2a57d7bc6bd781164062d6 WatchSource:0}: Error finding container e25540948ab73fd7bb0680b091e12c36f49aae8fda2a57d7bc6bd781164062d6: Status 404 returned error can't find the container with id e25540948ab73fd7bb0680b091e12c36f49aae8fda2a57d7bc6bd781164062d6 Jul 16 15:39:42.532811 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:42.532774 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" event={"ID":"6b29c0bf-6034-43e7-b04c-1dd7156bfe72","Type":"ContainerStarted","Data":"e25540948ab73fd7bb0680b091e12c36f49aae8fda2a57d7bc6bd781164062d6"} Jul 16 15:39:44.543891 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:44.543860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" event={"ID":"6b29c0bf-6034-43e7-b04c-1dd7156bfe72","Type":"ContainerStarted","Data":"1ada898abb8ca12a8ab7f116ff2aa5fecb34b1ff5e1f9b0cfa684d959d53429f"} Jul 16 15:39:44.561286 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:39:44.561244 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ggnfk" podStartSLOduration=1.007094502 podStartE2EDuration="3.561230225s" totalImagesPullingTime="2.554135723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:39:44.559744042 +0000 UTC m=+238.998270865" watchObservedRunningTime="2026-07-16 15:39:44.561230225 +0000 UTC m=+238.999757051" Jul 16 15:40:01.032342 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.032311 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd"] Jul 16 15:40:01.098548 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.098518 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.103031 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.103002 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 15:40:01.103333 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.103317 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 15:40:01.104150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.104112 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pnzkc\"" Jul 16 15:40:01.104150 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.104142 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 15:40:01.104393 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.104112 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 15:40:01.113540 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.113521 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd"] Jul 16 15:40:01.180931 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.180893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.181099 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.180935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.181099 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.181004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-tmp\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.181185 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.181102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-manifests\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.181185 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.181132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpx7\" (UniqueName: \"kubernetes.io/projected/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-kube-api-access-dtpx7\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.282532 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.282446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-manifests\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.282532 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.282482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpx7\" (UniqueName: \"kubernetes.io/projected/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-kube-api-access-dtpx7\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.282796 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.282560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.282796 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.282592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.282796 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.282647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-tmp\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.282943 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.282923 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-manifests\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.285000 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.284976 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-tmp\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.285214 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.285193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.285286 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.285193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.298526 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.298500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpx7\" (UniqueName: \"kubernetes.io/projected/3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef-kube-api-access-dtpx7\") pod \"opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd\" (UID: \"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.410303 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.410267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:01.570182 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.570154 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd"] Jul 16 15:40:01.572917 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:40:01.572892 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed48ac9_6e9a_40d1_adf9_3dae3c5683ef.slice/crio-7813a4a37aaf7a63d61148721f8818d1baeff01895337442526ceb1387957310 WatchSource:0}: Error finding container 7813a4a37aaf7a63d61148721f8818d1baeff01895337442526ceb1387957310: Status 404 returned error can't find the container with id 7813a4a37aaf7a63d61148721f8818d1baeff01895337442526ceb1387957310 Jul 16 15:40:01.613003 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:01.612968 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" event={"ID":"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef","Type":"ContainerStarted","Data":"7813a4a37aaf7a63d61148721f8818d1baeff01895337442526ceb1387957310"} Jul 16 15:40:05.632081 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:05.632049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" event={"ID":"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef","Type":"ContainerStarted","Data":"c7915794a573cf446541991a40717c21913017259d686afbba4e580b8d54d64a"} Jul 16 15:40:06.637296 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:06.637211 2573 generic.go:356] "Generic (PLEG): container finished" podID="3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef" containerID="c7915794a573cf446541991a40717c21913017259d686afbba4e580b8d54d64a" exitCode=0 Jul 16 15:40:06.637296 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:06.637258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" event={"ID":"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef","Type":"ContainerDied","Data":"c7915794a573cf446541991a40717c21913017259d686afbba4e580b8d54d64a"} Jul 16 15:40:07.643483 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:07.643408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" event={"ID":"3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef","Type":"ContainerStarted","Data":"9ae99ba2d6df26a66126422a9bcb24227921021949b6e284ad19053157638074"} Jul 16 15:40:07.643852 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:07.643525 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:07.667953 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:07.667892 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" podStartSLOduration=2.455872028 podStartE2EDuration="7.667871958s" totalImagesPullingTime="4.138086266s" totalInitContainerRuntime="1.073913664s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:40:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:40:07.66665532 +0000 UTC m=+262.105182144" watchObservedRunningTime="2026-07-16 15:40:07.667871958 +0000 UTC m=+262.106398782" Jul 16 15:40:18.650266 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:18.650236 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd" Jul 16 15:40:20.335854 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.335815 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk"] Jul 16 15:40:20.342939 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.342919 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.348088 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.348067 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 15:40:20.348200 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.348147 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 15:40:20.348200 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.348147 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6phh9\"" Jul 16 15:40:20.348316 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.348148 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 15:40:20.356451 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.356430 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk"] Jul 16 15:40:20.446864 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.446837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b8cdb65b-8aab-4819-85a2-64795faaa86d-manager-config\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.447027 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.446871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8cdb65b-8aab-4819-85a2-64795faaa86d-metrics-cert\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.447027 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.446890 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62h9c\" (UniqueName: \"kubernetes.io/projected/b8cdb65b-8aab-4819-85a2-64795faaa86d-kube-api-access-62h9c\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.447027 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.446985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8cdb65b-8aab-4819-85a2-64795faaa86d-cert\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.548169 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.548138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b8cdb65b-8aab-4819-85a2-64795faaa86d-manager-config\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.548306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.548175 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8cdb65b-8aab-4819-85a2-64795faaa86d-metrics-cert\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.548306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.548197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62h9c\" (UniqueName: \"kubernetes.io/projected/b8cdb65b-8aab-4819-85a2-64795faaa86d-kube-api-access-62h9c\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.548306 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.548240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8cdb65b-8aab-4819-85a2-64795faaa86d-cert\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.548795 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.548773 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b8cdb65b-8aab-4819-85a2-64795faaa86d-manager-config\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.550601 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.550580 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8cdb65b-8aab-4819-85a2-64795faaa86d-metrics-cert\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.550735 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.550718 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8cdb65b-8aab-4819-85a2-64795faaa86d-cert\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.556822 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.556796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62h9c\" (UniqueName: \"kubernetes.io/projected/b8cdb65b-8aab-4819-85a2-64795faaa86d-kube-api-access-62h9c\") pod \"lws-controller-manager-75498485d8-l6dnk\" (UID: \"b8cdb65b-8aab-4819-85a2-64795faaa86d\") " pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.652887 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.652811 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:20.781233 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:20.781206 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk"] Jul 16 15:40:20.783605 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:40:20.783580 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cdb65b_8aab_4819_85a2_64795faaa86d.slice/crio-791065f471d396d8305fb605d3eb912efeaab1a92ad84c02d1b4241954cfb99a WatchSource:0}: Error finding container 791065f471d396d8305fb605d3eb912efeaab1a92ad84c02d1b4241954cfb99a: Status 404 returned error can't find the container with id 791065f471d396d8305fb605d3eb912efeaab1a92ad84c02d1b4241954cfb99a Jul 16 15:40:21.707395 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:21.707357 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" event={"ID":"b8cdb65b-8aab-4819-85a2-64795faaa86d","Type":"ContainerStarted","Data":"791065f471d396d8305fb605d3eb912efeaab1a92ad84c02d1b4241954cfb99a"} Jul 16 15:40:23.182469 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.182433 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn"] Jul 16 15:40:23.186069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.186041 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.188677 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.188651 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 15:40:23.188829 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.188703 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 15:40:23.188829 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.188778 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gxtlg\"" Jul 16 15:40:23.189778 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.189734 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 15:40:23.189778 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.189774 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 15:40:23.197545 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.197522 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn"] Jul 16 15:40:23.273708 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.273667 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbt2q\" (UniqueName: \"kubernetes.io/projected/0f46714d-b3e2-461d-9c6d-5d053e2949a1-kube-api-access-pbt2q\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.273872 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.273781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f46714d-b3e2-461d-9c6d-5d053e2949a1-tmp\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.273916 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.273899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f46714d-b3e2-461d-9c6d-5d053e2949a1-tls-certs\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.375039 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.375000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f46714d-b3e2-461d-9c6d-5d053e2949a1-tls-certs\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.375208 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.375064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbt2q\" (UniqueName: \"kubernetes.io/projected/0f46714d-b3e2-461d-9c6d-5d053e2949a1-kube-api-access-pbt2q\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.375208 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.375109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f46714d-b3e2-461d-9c6d-5d053e2949a1-tmp\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.377435 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.377408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f46714d-b3e2-461d-9c6d-5d053e2949a1-tmp\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.377834 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.377809 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f46714d-b3e2-461d-9c6d-5d053e2949a1-tls-certs\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.392599 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.392573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbt2q\" (UniqueName: \"kubernetes.io/projected/0f46714d-b3e2-461d-9c6d-5d053e2949a1-kube-api-access-pbt2q\") pod \"kube-auth-proxy-5bb6d5565c-ccxjn\" (UID: \"0f46714d-b3e2-461d-9c6d-5d053e2949a1\") " pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.498455 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.498422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" Jul 16 15:40:23.632570 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.632546 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn"] Jul 16 15:40:23.634516 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:40:23.634488 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f46714d_b3e2_461d_9c6d_5d053e2949a1.slice/crio-0da469c10125e577186439427a0de1bb0ea47e16f5662eaa929dff51e66822ee WatchSource:0}: Error finding container 0da469c10125e577186439427a0de1bb0ea47e16f5662eaa929dff51e66822ee: Status 404 returned error can't find the container with id 0da469c10125e577186439427a0de1bb0ea47e16f5662eaa929dff51e66822ee Jul 16 15:40:23.717785 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.717752 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" event={"ID":"0f46714d-b3e2-461d-9c6d-5d053e2949a1","Type":"ContainerStarted","Data":"0da469c10125e577186439427a0de1bb0ea47e16f5662eaa929dff51e66822ee"} Jul 16 15:40:23.719138 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.719104 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" event={"ID":"b8cdb65b-8aab-4819-85a2-64795faaa86d","Type":"ContainerStarted","Data":"40107909bb24a30859b001c499c72a4223bd510f70657491e87eed35237a1545"} Jul 16 15:40:23.719318 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.719303 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:23.764387 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:23.764339 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" podStartSLOduration=1.56608509 podStartE2EDuration="3.764322067s" totalImagesPullingTime="2.198236977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:40:23.762839867 +0000 UTC m=+278.201366680" watchObservedRunningTime="2026-07-16 15:40:23.764322067 +0000 UTC m=+278.202848890" Jul 16 15:40:27.740733 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:27.740688 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" event={"ID":"0f46714d-b3e2-461d-9c6d-5d053e2949a1","Type":"ContainerStarted","Data":"7f1dbdac069cd7392384c386d6c54e11b1bc9158861e5b1bc728805d3866fa2e"} Jul 16 15:40:27.760647 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:27.760582 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bb6d5565c-ccxjn" podStartSLOduration=1.510422498 podStartE2EDuration="4.7605666s" totalImagesPullingTime="3.250144102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:40:27.759607841 +0000 UTC m=+282.198134665" watchObservedRunningTime="2026-07-16 15:40:27.7605666 +0000 UTC m=+282.199093424" Jul 16 15:40:34.697255 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.697219 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28"] Jul 16 15:40:34.701509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.701480 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:34.704224 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.704191 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 16 15:40:34.704356 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.704315 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-b6k87\"" Jul 16 15:40:34.710446 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.710424 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28"] Jul 16 15:40:34.725679 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.725659 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75498485d8-l6dnk" Jul 16 15:40:34.767371 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.767339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d556ecd-2925-441b-a755-61663894fb5d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w2b28\" (UID: \"1d556ecd-2925-441b-a755-61663894fb5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:34.767575 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.767517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt6mq\" (UniqueName: \"kubernetes.io/projected/1d556ecd-2925-441b-a755-61663894fb5d-kube-api-access-tt6mq\") pod \"llmisvc-controller-manager-777c6bd7d5-w2b28\" (UID: \"1d556ecd-2925-441b-a755-61663894fb5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:34.868379 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.868338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt6mq\" (UniqueName: \"kubernetes.io/projected/1d556ecd-2925-441b-a755-61663894fb5d-kube-api-access-tt6mq\") pod \"llmisvc-controller-manager-777c6bd7d5-w2b28\" (UID: \"1d556ecd-2925-441b-a755-61663894fb5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:34.868544 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.868398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d556ecd-2925-441b-a755-61663894fb5d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w2b28\" (UID: \"1d556ecd-2925-441b-a755-61663894fb5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:34.870815 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.870795 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d556ecd-2925-441b-a755-61663894fb5d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w2b28\" (UID: \"1d556ecd-2925-441b-a755-61663894fb5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:34.890547 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:34.890522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt6mq\" (UniqueName: \"kubernetes.io/projected/1d556ecd-2925-441b-a755-61663894fb5d-kube-api-access-tt6mq\") pod \"llmisvc-controller-manager-777c6bd7d5-w2b28\" (UID: \"1d556ecd-2925-441b-a755-61663894fb5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:35.013545 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:35.013512 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:35.148325 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:35.148302 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28"] Jul 16 15:40:35.150968 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:40:35.150935 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1d556ecd_2925_441b_a755_61663894fb5d.slice/crio-a745326505e767216612ecfc701547b783e7a595a1a30bc0b5cbb0ace1dc1f6f WatchSource:0}: Error finding container a745326505e767216612ecfc701547b783e7a595a1a30bc0b5cbb0ace1dc1f6f: Status 404 returned error can't find the container with id a745326505e767216612ecfc701547b783e7a595a1a30bc0b5cbb0ace1dc1f6f Jul 16 15:40:35.777724 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:35.777692 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" event={"ID":"1d556ecd-2925-441b-a755-61663894fb5d","Type":"ContainerStarted","Data":"a745326505e767216612ecfc701547b783e7a595a1a30bc0b5cbb0ace1dc1f6f"} Jul 16 15:40:38.794021 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:38.793988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" event={"ID":"1d556ecd-2925-441b-a755-61663894fb5d","Type":"ContainerStarted","Data":"020d6f93ed5aa2b94c3e0a7d5ba98d642af10ecf1b3282974ce301494f14a8bc"} Jul 16 15:40:38.794411 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:38.794118 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:40:38.816065 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:38.816012 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" podStartSLOduration=1.700223508 podStartE2EDuration="4.815997974s" totalImagesPullingTime="3.115774466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:40:38.81321126 +0000 UTC m=+293.251738082" watchObservedRunningTime="2026-07-16 15:40:38.815997974 +0000 UTC m=+293.254524808" Jul 16 15:40:45.980202 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:45.980178 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:40:45.982501 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:45.982480 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:40:45.983476 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:40:45.983458 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 15:41:09.800014 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:41:09.799983 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w2b28" Jul 16 15:42:13.976556 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.976525 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b6767c8c5-xvk2l"] Jul 16 15:42:13.979846 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.979828 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:13.983743 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.983717 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 15:42:13.983911 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.983737 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hr47v\"" Jul 16 15:42:13.983911 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.983744 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 15:42:13.984075 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.983789 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 15:42:13.984075 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.983788 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 15:42:13.984075 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.983796 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 15:42:13.984075 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.983801 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 15:42:13.984075 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.983856 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 15:42:13.988295 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.988276 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 15:42:13.992315 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:13.992294 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6767c8c5-xvk2l"] Jul 16 15:42:14.053827 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.053804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-service-ca\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.053946 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.053835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-console-config\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.053946 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.053861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxlj\" (UniqueName: \"kubernetes.io/projected/0e453bb7-307f-4733-870d-f1035340b3b9-kube-api-access-xlxlj\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.053946 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.053924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e453bb7-307f-4733-870d-f1035340b3b9-console-oauth-config\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.053946 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.053942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-oauth-serving-cert\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.054144 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.054010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e453bb7-307f-4733-870d-f1035340b3b9-console-serving-cert\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.054144 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.054045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-trusted-ca-bundle\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.154637 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.154579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-console-config\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.154794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.154659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxlj\" (UniqueName: \"kubernetes.io/projected/0e453bb7-307f-4733-870d-f1035340b3b9-kube-api-access-xlxlj\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.154794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.154715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e453bb7-307f-4733-870d-f1035340b3b9-console-oauth-config\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.154794 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.154742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-oauth-serving-cert\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.154973 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.154803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e453bb7-307f-4733-870d-f1035340b3b9-console-serving-cert\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.154973 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.154839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-trusted-ca-bundle\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.154973 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.154884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-service-ca\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.155441 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.155413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-oauth-serving-cert\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.155571 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.155413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-console-config\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.155571 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.155540 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-service-ca\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.155706 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.155630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e453bb7-307f-4733-870d-f1035340b3b9-trusted-ca-bundle\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.157233 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.157205 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e453bb7-307f-4733-870d-f1035340b3b9-console-serving-cert\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.157374 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.157356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e453bb7-307f-4733-870d-f1035340b3b9-console-oauth-config\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.163360 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.163342 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxlj\" (UniqueName: \"kubernetes.io/projected/0e453bb7-307f-4733-870d-f1035340b3b9-kube-api-access-xlxlj\") pod \"console-6b6767c8c5-xvk2l\" (UID: \"0e453bb7-307f-4733-870d-f1035340b3b9\") " pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.290432 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.290405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:14.420287 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:14.420262 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6767c8c5-xvk2l"] Jul 16 15:42:14.422650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:42:14.422604 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e453bb7_307f_4733_870d_f1035340b3b9.slice/crio-9c8eb27f7bb1e2eac6e0074a62265820a41f69f4c8446d5f8da3f044c315f010 WatchSource:0}: Error finding container 9c8eb27f7bb1e2eac6e0074a62265820a41f69f4c8446d5f8da3f044c315f010: Status 404 returned error can't find the container with id 9c8eb27f7bb1e2eac6e0074a62265820a41f69f4c8446d5f8da3f044c315f010 Jul 16 15:42:15.233989 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:15.233956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6767c8c5-xvk2l" event={"ID":"0e453bb7-307f-4733-870d-f1035340b3b9","Type":"ContainerStarted","Data":"c58a26fae588906f60e8312c63e19b4aa087afc6017e72ec4761150d1a9663a9"} Jul 16 15:42:15.234333 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:15.234005 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6767c8c5-xvk2l" event={"ID":"0e453bb7-307f-4733-870d-f1035340b3b9","Type":"ContainerStarted","Data":"9c8eb27f7bb1e2eac6e0074a62265820a41f69f4c8446d5f8da3f044c315f010"} Jul 16 15:42:15.261633 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:15.261577 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b6767c8c5-xvk2l" podStartSLOduration=2.2615606489999998 podStartE2EDuration="2.261560649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:42:15.259259159 +0000 UTC m=+389.697785974" watchObservedRunningTime="2026-07-16 15:42:15.261560649 +0000 UTC m=+389.700087471" Jul 16 15:42:24.291009 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:24.290970 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:24.291471 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:24.291052 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:24.295766 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:24.295746 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:25.286028 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:25.285992 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b6767c8c5-xvk2l" Jul 16 15:42:28.469252 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.469224 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f"] Jul 16 15:42:28.472671 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.472654 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:28.477354 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.477328 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 15:42:28.477494 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.477479 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nlqrf\"" Jul 16 15:42:28.480129 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.480105 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 15:42:28.521205 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.521175 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f"] Jul 16 15:42:28.582443 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.582415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s4z4f\" (UID: \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:28.582564 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.582477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmqjj\" (UniqueName: \"kubernetes.io/projected/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-kube-api-access-zmqjj\") pod \"kuadrant-operator-controller-manager-86878c769f-s4z4f\" (UID: \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:28.683610 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.683577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s4z4f\" (UID: \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:28.683756 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.683688 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmqjj\" (UniqueName: \"kubernetes.io/projected/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-kube-api-access-zmqjj\") pod \"kuadrant-operator-controller-manager-86878c769f-s4z4f\" (UID: \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:28.683955 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.683937 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s4z4f\" (UID: \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:28.706952 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.706924 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmqjj\" (UniqueName: \"kubernetes.io/projected/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-kube-api-access-zmqjj\") pod \"kuadrant-operator-controller-manager-86878c769f-s4z4f\" (UID: \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:28.782927 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.782894 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:28.963385 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.963362 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f"] Jul 16 15:42:28.966629 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:42:28.966582 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7bfd0a_cf3f_4bdb_908e_f4a5e5b561cc.slice/crio-77b5de1d995a94911dd712118bcf7c53cee05e5a914673f0a1e52495e53f4442 WatchSource:0}: Error finding container 77b5de1d995a94911dd712118bcf7c53cee05e5a914673f0a1e52495e53f4442: Status 404 returned error can't find the container with id 77b5de1d995a94911dd712118bcf7c53cee05e5a914673f0a1e52495e53f4442 Jul 16 15:42:28.969784 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:28.969768 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 15:42:29.301196 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:29.301156 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" event={"ID":"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc","Type":"ContainerStarted","Data":"77b5de1d995a94911dd712118bcf7c53cee05e5a914673f0a1e52495e53f4442"} Jul 16 15:42:36.338819 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:36.338773 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" event={"ID":"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc","Type":"ContainerStarted","Data":"8b8c8697a5ef3473a2dd6fdaea467dcf73f604961dcfaa786b141867b9eb7352"} Jul 16 15:42:36.339313 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:36.338834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:42:36.364877 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:36.364804 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" podStartSLOduration=1.967270844 podStartE2EDuration="8.3647905s" totalImagesPullingTime="6.397519656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:42:36.362011839 +0000 UTC m=+410.800538663" watchObservedRunningTime="2026-07-16 15:42:36.3647905 +0000 UTC m=+410.803317322" Jul 16 15:42:47.345569 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:42:47.345538 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:43:36.232722 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.232648 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8586d78ff7-hpqbd"] Jul 16 15:43:36.235509 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.235493 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" Jul 16 15:43:36.238657 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.238636 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lg7h5\"" Jul 16 15:43:36.243655 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.243632 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8586d78ff7-hpqbd"] Jul 16 15:43:36.287381 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.287355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczmv\" (UniqueName: \"kubernetes.io/projected/ec263ff4-9034-48bd-b0a9-4efb456aaeec-kube-api-access-fczmv\") pod \"authorino-8586d78ff7-hpqbd\" (UID: \"ec263ff4-9034-48bd-b0a9-4efb456aaeec\") " pod="kuadrant-system/authorino-8586d78ff7-hpqbd" Jul 16 15:43:36.387976 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.387949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczmv\" (UniqueName: \"kubernetes.io/projected/ec263ff4-9034-48bd-b0a9-4efb456aaeec-kube-api-access-fczmv\") pod \"authorino-8586d78ff7-hpqbd\" (UID: \"ec263ff4-9034-48bd-b0a9-4efb456aaeec\") " pod="kuadrant-system/authorino-8586d78ff7-hpqbd" Jul 16 15:43:36.397145 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.397112 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczmv\" (UniqueName: \"kubernetes.io/projected/ec263ff4-9034-48bd-b0a9-4efb456aaeec-kube-api-access-fczmv\") pod \"authorino-8586d78ff7-hpqbd\" (UID: \"ec263ff4-9034-48bd-b0a9-4efb456aaeec\") " pod="kuadrant-system/authorino-8586d78ff7-hpqbd" Jul 16 15:43:36.457018 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.456992 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8586d78ff7-hpqbd"] Jul 16 15:43:36.457245 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.457233 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" Jul 16 15:43:36.587310 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.587277 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8586d78ff7-hpqbd"] Jul 16 15:43:36.591013 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:43:36.590991 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec263ff4_9034_48bd_b0a9_4efb456aaeec.slice/crio-331e8aa07201c97c07bc227f7676a44bc5dc44f08027f145dbfb435edc5a4c95 WatchSource:0}: Error finding container 331e8aa07201c97c07bc227f7676a44bc5dc44f08027f145dbfb435edc5a4c95: Status 404 returned error can't find the container with id 331e8aa07201c97c07bc227f7676a44bc5dc44f08027f145dbfb435edc5a4c95 Jul 16 15:43:36.618598 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:36.618573 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" event={"ID":"ec263ff4-9034-48bd-b0a9-4efb456aaeec","Type":"ContainerStarted","Data":"331e8aa07201c97c07bc227f7676a44bc5dc44f08027f145dbfb435edc5a4c95"} Jul 16 15:43:40.643554 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:40.643516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" event={"ID":"ec263ff4-9034-48bd-b0a9-4efb456aaeec","Type":"ContainerStarted","Data":"ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25"} Jul 16 15:43:40.643991 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:40.643555 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" podUID="ec263ff4-9034-48bd-b0a9-4efb456aaeec" containerName="authorino" containerID="cri-o://ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25" gracePeriod=30 Jul 16 15:43:40.661916 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:40.661873 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" podStartSLOduration=1.593121384 podStartE2EDuration="4.661858132s" totalImagesPullingTime="3.068736748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:43:40.66030167 +0000 UTC m=+475.098828494" watchObservedRunningTime="2026-07-16 15:43:40.661858132 +0000 UTC m=+475.100384954" Jul 16 15:43:40.883039 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:40.883013 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" Jul 16 15:43:40.934419 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:40.934357 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fczmv\" (UniqueName: \"kubernetes.io/projected/ec263ff4-9034-48bd-b0a9-4efb456aaeec-kube-api-access-fczmv\") pod \"ec263ff4-9034-48bd-b0a9-4efb456aaeec\" (UID: \"ec263ff4-9034-48bd-b0a9-4efb456aaeec\") " Jul 16 15:43:40.936360 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:40.936331 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec263ff4-9034-48bd-b0a9-4efb456aaeec-kube-api-access-fczmv" (OuterVolumeSpecName: "kube-api-access-fczmv") pod "ec263ff4-9034-48bd-b0a9-4efb456aaeec" (UID: "ec263ff4-9034-48bd-b0a9-4efb456aaeec"). InnerVolumeSpecName "kube-api-access-fczmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:43:41.035549 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.035522 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fczmv\" (UniqueName: \"kubernetes.io/projected/ec263ff4-9034-48bd-b0a9-4efb456aaeec-kube-api-access-fczmv\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:43:41.649014 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.648981 2573 generic.go:356] "Generic (PLEG): container finished" podID="ec263ff4-9034-48bd-b0a9-4efb456aaeec" containerID="ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25" exitCode=0 Jul 16 15:43:41.649847 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.649028 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" Jul 16 15:43:41.649847 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.649052 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" event={"ID":"ec263ff4-9034-48bd-b0a9-4efb456aaeec","Type":"ContainerDied","Data":"ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25"} Jul 16 15:43:41.649847 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.649086 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8586d78ff7-hpqbd" event={"ID":"ec263ff4-9034-48bd-b0a9-4efb456aaeec","Type":"ContainerDied","Data":"331e8aa07201c97c07bc227f7676a44bc5dc44f08027f145dbfb435edc5a4c95"} Jul 16 15:43:41.649847 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.649107 2573 scope.go:117] "RemoveContainer" containerID="ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25" Jul 16 15:43:41.657345 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.657318 2573 scope.go:117] "RemoveContainer" containerID="ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25" Jul 16 15:43:41.657574 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:43:41.657556 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25\": container with ID starting with ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25 not found: ID does not exist" containerID="ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25" Jul 16 15:43:41.657661 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.657582 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25"} err="failed to get container status \"ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25\": rpc error: code = NotFound desc = could not find container \"ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25\": container with ID starting with ea8bc086229f32549df7e11f04b39cbdb25789ea4af3e11bdaa4a65a4e536f25 not found: ID does not exist" Jul 16 15:43:41.672070 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.672040 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8586d78ff7-hpqbd"] Jul 16 15:43:41.678102 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:41.678074 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8586d78ff7-hpqbd"] Jul 16 15:43:42.385243 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:42.385210 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec263ff4-9034-48bd-b0a9-4efb456aaeec" path="/var/lib/kubelet/pods/ec263ff4-9034-48bd-b0a9-4efb456aaeec/volumes" Jul 16 15:43:43.967944 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.967906 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75fbd6b86b-fd54p"] Jul 16 15:43:43.968473 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.968453 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec263ff4-9034-48bd-b0a9-4efb456aaeec" containerName="authorino" Jul 16 15:43:43.968553 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.968477 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec263ff4-9034-48bd-b0a9-4efb456aaeec" containerName="authorino" Jul 16 15:43:43.968607 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.968560 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec263ff4-9034-48bd-b0a9-4efb456aaeec" containerName="authorino" Jul 16 15:43:43.971881 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.971859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:43.974695 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.974664 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 16 15:43:43.974695 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.974674 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 15:43:43.974874 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.974773 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lf24x\"" Jul 16 15:43:43.982988 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:43.982964 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75fbd6b86b-fd54p"] Jul 16 15:43:44.067124 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:44.067101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgscw\" (UniqueName: \"kubernetes.io/projected/f1cb18f9-81fe-46b1-a228-60f571163946-kube-api-access-lgscw\") pod \"maas-controller-75fbd6b86b-fd54p\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:44.067249 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:44.067164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert\") pod \"maas-controller-75fbd6b86b-fd54p\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:44.168032 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:44.168002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert\") pod \"maas-controller-75fbd6b86b-fd54p\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:44.168152 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:44.168118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgscw\" (UniqueName: \"kubernetes.io/projected/f1cb18f9-81fe-46b1-a228-60f571163946-kube-api-access-lgscw\") pod \"maas-controller-75fbd6b86b-fd54p\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:44.168193 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:43:44.168153 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 16 15:43:44.168235 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:43:44.168224 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert podName:f1cb18f9-81fe-46b1-a228-60f571163946 nodeName:}" failed. No retries permitted until 2026-07-16 15:43:44.668204026 +0000 UTC m=+479.106730830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert") pod "maas-controller-75fbd6b86b-fd54p" (UID: "f1cb18f9-81fe-46b1-a228-60f571163946") : secret "maas-controller-webhook-cert" not found Jul 16 15:43:44.177319 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:44.177296 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgscw\" (UniqueName: \"kubernetes.io/projected/f1cb18f9-81fe-46b1-a228-60f571163946-kube-api-access-lgscw\") pod \"maas-controller-75fbd6b86b-fd54p\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:44.673443 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:44.673410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert\") pod \"maas-controller-75fbd6b86b-fd54p\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:44.675767 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:44.675748 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert\") pod \"maas-controller-75fbd6b86b-fd54p\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:44.882755 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:44.882717 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:45.012032 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:45.012002 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75fbd6b86b-fd54p"] Jul 16 15:43:45.013897 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:43:45.013860 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cb18f9_81fe_46b1_a228_60f571163946.slice/crio-abe850e91b4bdbf7e5e2ab1696b5359269ecd4a048e58a9d21327cc831453400 WatchSource:0}: Error finding container abe850e91b4bdbf7e5e2ab1696b5359269ecd4a048e58a9d21327cc831453400: Status 404 returned error can't find the container with id abe850e91b4bdbf7e5e2ab1696b5359269ecd4a048e58a9d21327cc831453400 Jul 16 15:43:45.676528 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:45.676490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" event={"ID":"f1cb18f9-81fe-46b1-a228-60f571163946","Type":"ContainerStarted","Data":"abe850e91b4bdbf7e5e2ab1696b5359269ecd4a048e58a9d21327cc831453400"} Jul 16 15:43:47.440399 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:47.440372 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 15:43:47.689427 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:47.689331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" event={"ID":"f1cb18f9-81fe-46b1-a228-60f571163946","Type":"ContainerStarted","Data":"f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61"} Jul 16 15:43:47.689427 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:47.689400 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:43:47.713865 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:47.713819 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" podStartSLOduration=2.291754622 podStartE2EDuration="4.713797795s" totalImagesPullingTime="2.422043173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:43:47.712760366 +0000 UTC m=+482.151287192" watchObservedRunningTime="2026-07-16 15:43:47.713797795 +0000 UTC m=+482.152324616" Jul 16 15:43:58.700477 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:43:58.700442 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:44:00.403778 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.403742 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75fbd6b86b-fd54p"] Jul 16 15:44:00.404152 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.403908 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" podUID="f1cb18f9-81fe-46b1-a228-60f571163946" containerName="manager" containerID="cri-o://f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61" gracePeriod=30 Jul 16 15:44:00.643030 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.643006 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:44:00.753272 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.753182 2573 generic.go:356] "Generic (PLEG): container finished" podID="f1cb18f9-81fe-46b1-a228-60f571163946" containerID="f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61" exitCode=0 Jul 16 15:44:00.753272 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.753230 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" Jul 16 15:44:00.753499 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.753273 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" event={"ID":"f1cb18f9-81fe-46b1-a228-60f571163946","Type":"ContainerDied","Data":"f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61"} Jul 16 15:44:00.753499 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.753337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75fbd6b86b-fd54p" event={"ID":"f1cb18f9-81fe-46b1-a228-60f571163946","Type":"ContainerDied","Data":"abe850e91b4bdbf7e5e2ab1696b5359269ecd4a048e58a9d21327cc831453400"} Jul 16 15:44:00.753499 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.753366 2573 scope.go:117] "RemoveContainer" containerID="f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61" Jul 16 15:44:00.761731 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.761704 2573 scope.go:117] "RemoveContainer" containerID="f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61" Jul 16 15:44:00.761992 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:44:00.761973 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61\": container with ID starting with f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61 not found: ID does not exist" containerID="f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61" Jul 16 15:44:00.762053 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.762003 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61"} err="failed to get container status \"f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61\": rpc error: code = NotFound desc = could not find container \"f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61\": container with ID starting with f1354c87242f54e62a28574b9d5a46118f43c350b0f26a67678fcf8245dfeb61 not found: ID does not exist" Jul 16 15:44:00.822349 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.822327 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert\") pod \"f1cb18f9-81fe-46b1-a228-60f571163946\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " Jul 16 15:44:00.822447 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.822378 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgscw\" (UniqueName: \"kubernetes.io/projected/f1cb18f9-81fe-46b1-a228-60f571163946-kube-api-access-lgscw\") pod \"f1cb18f9-81fe-46b1-a228-60f571163946\" (UID: \"f1cb18f9-81fe-46b1-a228-60f571163946\") " Jul 16 15:44:00.824456 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.824423 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f1cb18f9-81fe-46b1-a228-60f571163946" (UID: "f1cb18f9-81fe-46b1-a228-60f571163946"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:44:00.824456 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.824426 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cb18f9-81fe-46b1-a228-60f571163946-kube-api-access-lgscw" (OuterVolumeSpecName: "kube-api-access-lgscw") pod "f1cb18f9-81fe-46b1-a228-60f571163946" (UID: "f1cb18f9-81fe-46b1-a228-60f571163946"). InnerVolumeSpecName "kube-api-access-lgscw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:44:00.922988 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.922967 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cb18f9-81fe-46b1-a228-60f571163946-webhook-cert\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:44:00.922988 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:00.922989 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgscw\" (UniqueName: \"kubernetes.io/projected/f1cb18f9-81fe-46b1-a228-60f571163946-kube-api-access-lgscw\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:44:01.076685 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:01.076655 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75fbd6b86b-fd54p"] Jul 16 15:44:01.080587 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:01.080563 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75fbd6b86b-fd54p"] Jul 16 15:44:02.383553 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:02.383517 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1cb18f9-81fe-46b1-a228-60f571163946" path="/var/lib/kubelet/pods/f1cb18f9-81fe-46b1-a228-60f571163946/volumes" Jul 16 15:44:20.484953 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.484921 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn"] Jul 16 15:44:20.485393 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.485313 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1cb18f9-81fe-46b1-a228-60f571163946" containerName="manager" Jul 16 15:44:20.485393 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.485324 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1cb18f9-81fe-46b1-a228-60f571163946" containerName="manager" Jul 16 15:44:20.485393 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.485389 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1cb18f9-81fe-46b1-a228-60f571163946" containerName="manager" Jul 16 15:44:20.495375 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.495347 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.498093 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.498069 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 16 15:44:20.498289 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.498266 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 15:44:20.498386 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.498210 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 15:44:20.498531 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.498484 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwrlz\"" Jul 16 15:44:20.499709 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.499686 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn"] Jul 16 15:44:20.602937 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.602898 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b1e242-4d62-49dc-9a68-7d8aa41cc225-openshift-service-ca\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.602937 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.602947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b1e242-4d62-49dc-9a68-7d8aa41cc225-maas-api-tls\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.603130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.603019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84wz\" (UniqueName: \"kubernetes.io/projected/57b1e242-4d62-49dc-9a68-7d8aa41cc225-kube-api-access-v84wz\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.703625 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.703580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b1e242-4d62-49dc-9a68-7d8aa41cc225-openshift-service-ca\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.703790 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.703641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b1e242-4d62-49dc-9a68-7d8aa41cc225-maas-api-tls\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.703790 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.703697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84wz\" (UniqueName: \"kubernetes.io/projected/57b1e242-4d62-49dc-9a68-7d8aa41cc225-kube-api-access-v84wz\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.704224 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.704204 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b1e242-4d62-49dc-9a68-7d8aa41cc225-openshift-service-ca\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.706130 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.706109 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b1e242-4d62-49dc-9a68-7d8aa41cc225-maas-api-tls\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.713779 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.713751 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84wz\" (UniqueName: \"kubernetes.io/projected/57b1e242-4d62-49dc-9a68-7d8aa41cc225-kube-api-access-v84wz\") pod \"maas-api-5b6d86c5cf-66pgn\" (UID: \"57b1e242-4d62-49dc-9a68-7d8aa41cc225\") " pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.806826 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.806784 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:20.948329 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:20.948302 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn"] Jul 16 15:44:20.950608 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:44:20.950582 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b1e242_4d62_49dc_9a68_7d8aa41cc225.slice/crio-81d39b4bdee8b3f889adb8aa274a90f28631713e16bd2ea767d591cf397a27fb WatchSource:0}: Error finding container 81d39b4bdee8b3f889adb8aa274a90f28631713e16bd2ea767d591cf397a27fb: Status 404 returned error can't find the container with id 81d39b4bdee8b3f889adb8aa274a90f28631713e16bd2ea767d591cf397a27fb Jul 16 15:44:21.853853 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:21.853804 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" event={"ID":"57b1e242-4d62-49dc-9a68-7d8aa41cc225","Type":"ContainerStarted","Data":"81d39b4bdee8b3f889adb8aa274a90f28631713e16bd2ea767d591cf397a27fb"} Jul 16 15:44:22.860307 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:22.860277 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" event={"ID":"57b1e242-4d62-49dc-9a68-7d8aa41cc225","Type":"ContainerStarted","Data":"d8f040c50f13dac1aa2a90e9faba2dbbcfcf50b39af9399740ad1e15a7452cb2"} Jul 16 15:44:22.860706 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:22.860406 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:44:22.879456 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:22.879412 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" podStartSLOduration=1.480272565 podStartE2EDuration="2.879399043s" totalImagesPullingTime="1.399126478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:44:22.878062346 +0000 UTC m=+517.316589175" watchObservedRunningTime="2026-07-16 15:44:22.879399043 +0000 UTC m=+517.317925866" Jul 16 15:44:28.871626 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:44:28.871593 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b6d86c5cf-66pgn" Jul 16 15:45:00.146176 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:00.146142 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv"] Jul 16 15:45:00.149724 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:00.149703 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" Jul 16 15:45:00.159412 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:00.159384 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv"] Jul 16 15:45:00.266887 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:00.266853 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtdbn\" (UniqueName: \"kubernetes.io/projected/794a5ef0-9c48-4951-a298-d7586d574d43-kube-api-access-jtdbn\") pod \"maas-api-key-cleanup-29736945-chswv\" (UID: \"794a5ef0-9c48-4951-a298-d7586d574d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" Jul 16 15:45:00.368304 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:00.368267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtdbn\" (UniqueName: \"kubernetes.io/projected/794a5ef0-9c48-4951-a298-d7586d574d43-kube-api-access-jtdbn\") pod \"maas-api-key-cleanup-29736945-chswv\" (UID: \"794a5ef0-9c48-4951-a298-d7586d574d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" Jul 16 15:45:00.380944 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:00.380918 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtdbn\" (UniqueName: \"kubernetes.io/projected/794a5ef0-9c48-4951-a298-d7586d574d43-kube-api-access-jtdbn\") pod \"maas-api-key-cleanup-29736945-chswv\" (UID: \"794a5ef0-9c48-4951-a298-d7586d574d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" Jul 16 15:45:00.463184 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:00.463117 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" Jul 16 15:45:00.588915 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:00.588887 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv"] Jul 16 15:45:00.591629 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:45:00.591588 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794a5ef0_9c48_4951_a298_d7586d574d43.slice/crio-c19e9717b3aaccbc4d8fda0f5742ba60baa2b143da06030ced3d87b751eb6c01 WatchSource:0}: Error finding container c19e9717b3aaccbc4d8fda0f5742ba60baa2b143da06030ced3d87b751eb6c01: Status 404 returned error can't find the container with id c19e9717b3aaccbc4d8fda0f5742ba60baa2b143da06030ced3d87b751eb6c01 Jul 16 15:45:01.048768 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:01.048736 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" event={"ID":"794a5ef0-9c48-4951-a298-d7586d574d43","Type":"ContainerStarted","Data":"c19e9717b3aaccbc4d8fda0f5742ba60baa2b143da06030ced3d87b751eb6c01"} Jul 16 15:45:04.067649 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:04.067593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" event={"ID":"794a5ef0-9c48-4951-a298-d7586d574d43","Type":"ContainerStarted","Data":"df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa"} Jul 16 15:45:04.088487 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:04.088428 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" podStartSLOduration=1.515028331 podStartE2EDuration="4.088408661s" totalImagesPullingTime="2.57338033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:45:04.087985922 +0000 UTC m=+558.526512758" watchObservedRunningTime="2026-07-16 15:45:04.088408661 +0000 UTC m=+558.526935488" Jul 16 15:45:46.013022 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:46.012997 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:45:46.013465 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:45:46.013006 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:46:00.010561 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:00.010527 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv"] Jul 16 15:46:00.011047 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:00.010715 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" podUID="794a5ef0-9c48-4951-a298-d7586d574d43" containerName="cleanup" containerID="cri-o://df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa" gracePeriod=30 Jul 16 15:46:09.242468 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.242377 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66647745d4-t7vz7"] Jul 16 15:46:09.246270 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.246246 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66647745d4-t7vz7" Jul 16 15:46:09.249982 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.249959 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lg7h5\"" Jul 16 15:46:09.250086 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.249969 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 15:46:09.258407 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.258331 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66647745d4-t7vz7"] Jul 16 15:46:09.306959 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.306925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61-tls-cert\") pod \"authorino-66647745d4-t7vz7\" (UID: \"92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61\") " pod="kuadrant-system/authorino-66647745d4-t7vz7" Jul 16 15:46:09.307096 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.306971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvr2q\" (UniqueName: \"kubernetes.io/projected/92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61-kube-api-access-xvr2q\") pod \"authorino-66647745d4-t7vz7\" (UID: \"92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61\") " pod="kuadrant-system/authorino-66647745d4-t7vz7" Jul 16 15:46:09.408287 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.408253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61-tls-cert\") pod \"authorino-66647745d4-t7vz7\" (UID: \"92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61\") " pod="kuadrant-system/authorino-66647745d4-t7vz7" Jul 16 15:46:09.408454 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.408304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvr2q\" (UniqueName: \"kubernetes.io/projected/92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61-kube-api-access-xvr2q\") pod \"authorino-66647745d4-t7vz7\" (UID: \"92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61\") " pod="kuadrant-system/authorino-66647745d4-t7vz7" Jul 16 15:46:09.410796 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.410764 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61-tls-cert\") pod \"authorino-66647745d4-t7vz7\" (UID: \"92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61\") " pod="kuadrant-system/authorino-66647745d4-t7vz7" Jul 16 15:46:09.418369 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.418345 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvr2q\" (UniqueName: \"kubernetes.io/projected/92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61-kube-api-access-xvr2q\") pod \"authorino-66647745d4-t7vz7\" (UID: \"92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61\") " pod="kuadrant-system/authorino-66647745d4-t7vz7" Jul 16 15:46:09.555840 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.555805 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66647745d4-t7vz7" Jul 16 15:46:09.687686 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:09.687662 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66647745d4-t7vz7"] Jul 16 15:46:09.689650 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:46:09.689598 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92a7ebd8_bcbf_4ed7_8716_4c5bd694dd61.slice/crio-12b74293c352ae6edd878c29f44561c1b63d6e18ce53f365353d240d326852ab WatchSource:0}: Error finding container 12b74293c352ae6edd878c29f44561c1b63d6e18ce53f365353d240d326852ab: Status 404 returned error can't find the container with id 12b74293c352ae6edd878c29f44561c1b63d6e18ce53f365353d240d326852ab Jul 16 15:46:10.384205 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:10.384118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66647745d4-t7vz7" event={"ID":"92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61","Type":"ContainerStarted","Data":"fbda1cb52105bdb4f0b8278dbf5b6ea900f0cf5110780940f6f1994416ca7aa3"} Jul 16 15:46:10.384205 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:10.384169 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66647745d4-t7vz7" event={"ID":"92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61","Type":"ContainerStarted","Data":"12b74293c352ae6edd878c29f44561c1b63d6e18ce53f365353d240d326852ab"} Jul 16 15:46:10.404952 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:10.404903 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66647745d4-t7vz7" podStartSLOduration=1.056546211 podStartE2EDuration="1.404886278s" totalImagesPullingTime="348.340067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:46:10.402818044 +0000 UTC m=+624.841344867" watchObservedRunningTime="2026-07-16 15:46:10.404886278 +0000 UTC m=+624.843413160" Jul 16 15:46:30.182890 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.182858 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" Jul 16 15:46:30.308106 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.308031 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtdbn\" (UniqueName: \"kubernetes.io/projected/794a5ef0-9c48-4951-a298-d7586d574d43-kube-api-access-jtdbn\") pod \"794a5ef0-9c48-4951-a298-d7586d574d43\" (UID: \"794a5ef0-9c48-4951-a298-d7586d574d43\") " Jul 16 15:46:30.310252 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.310225 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794a5ef0-9c48-4951-a298-d7586d574d43-kube-api-access-jtdbn" (OuterVolumeSpecName: "kube-api-access-jtdbn") pod "794a5ef0-9c48-4951-a298-d7586d574d43" (UID: "794a5ef0-9c48-4951-a298-d7586d574d43"). InnerVolumeSpecName "kube-api-access-jtdbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:46:30.409431 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.409403 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtdbn\" (UniqueName: \"kubernetes.io/projected/794a5ef0-9c48-4951-a298-d7586d574d43-kube-api-access-jtdbn\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:46:30.491969 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.491935 2573 generic.go:356] "Generic (PLEG): container finished" podID="794a5ef0-9c48-4951-a298-d7586d574d43" containerID="df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa" exitCode=137 Jul 16 15:46:30.492107 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.491997 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" Jul 16 15:46:30.492107 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.492024 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" event={"ID":"794a5ef0-9c48-4951-a298-d7586d574d43","Type":"ContainerDied","Data":"df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa"} Jul 16 15:46:30.492107 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.492074 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv" event={"ID":"794a5ef0-9c48-4951-a298-d7586d574d43","Type":"ContainerDied","Data":"c19e9717b3aaccbc4d8fda0f5742ba60baa2b143da06030ced3d87b751eb6c01"} Jul 16 15:46:30.492107 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.492096 2573 scope.go:117] "RemoveContainer" containerID="df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa" Jul 16 15:46:30.500725 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.500702 2573 scope.go:117] "RemoveContainer" containerID="df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa" Jul 16 15:46:30.500964 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:46:30.500941 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa\": container with ID starting with df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa not found: ID does not exist" containerID="df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa" Jul 16 15:46:30.501053 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.500976 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa"} err="failed to get container status \"df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa\": rpc error: code = NotFound desc = could not find container \"df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa\": container with ID starting with df019ec6d22289d15112d1657f3c3e831068732dcd367f8986ae8036923e6faa not found: ID does not exist" Jul 16 15:46:30.510165 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.510146 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv"] Jul 16 15:46:30.514476 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:30.514456 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736945-chswv"] Jul 16 15:46:32.383731 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:46:32.383699 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="794a5ef0-9c48-4951-a298-d7586d574d43" path="/var/lib/kubelet/pods/794a5ef0-9c48-4951-a298-d7586d574d43/volumes" Jul 16 15:47:29.369515 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.369477 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-775f5d4dfb-m6fdc"] Jul 16 15:47:29.370054 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.369914 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794a5ef0-9c48-4951-a298-d7586d574d43" containerName="cleanup" Jul 16 15:47:29.370054 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.369926 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="794a5ef0-9c48-4951-a298-d7586d574d43" containerName="cleanup" Jul 16 15:47:29.370054 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.370015 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="794a5ef0-9c48-4951-a298-d7586d574d43" containerName="cleanup" Jul 16 15:47:29.373048 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.373032 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:29.384353 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.384332 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 15:47:29.384592 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.384578 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 16 15:47:29.386427 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.386413 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lf24x\"" Jul 16 15:47:29.395095 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.395068 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-775f5d4dfb-m6fdc"] Jul 16 15:47:29.456869 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.456837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmbz\" (UniqueName: \"kubernetes.io/projected/a4eff9f4-e102-42b7-b33f-6f798dadb4e1-kube-api-access-2zmbz\") pod \"maas-controller-775f5d4dfb-m6fdc\" (UID: \"a4eff9f4-e102-42b7-b33f-6f798dadb4e1\") " pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:29.457021 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.456982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4eff9f4-e102-42b7-b33f-6f798dadb4e1-webhook-cert\") pod \"maas-controller-775f5d4dfb-m6fdc\" (UID: \"a4eff9f4-e102-42b7-b33f-6f798dadb4e1\") " pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:29.557693 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.557668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4eff9f4-e102-42b7-b33f-6f798dadb4e1-webhook-cert\") pod \"maas-controller-775f5d4dfb-m6fdc\" (UID: \"a4eff9f4-e102-42b7-b33f-6f798dadb4e1\") " pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:29.557843 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.557713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmbz\" (UniqueName: \"kubernetes.io/projected/a4eff9f4-e102-42b7-b33f-6f798dadb4e1-kube-api-access-2zmbz\") pod \"maas-controller-775f5d4dfb-m6fdc\" (UID: \"a4eff9f4-e102-42b7-b33f-6f798dadb4e1\") " pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:29.560155 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.560131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4eff9f4-e102-42b7-b33f-6f798dadb4e1-webhook-cert\") pod \"maas-controller-775f5d4dfb-m6fdc\" (UID: \"a4eff9f4-e102-42b7-b33f-6f798dadb4e1\") " pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:29.570044 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.570018 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmbz\" (UniqueName: \"kubernetes.io/projected/a4eff9f4-e102-42b7-b33f-6f798dadb4e1-kube-api-access-2zmbz\") pod \"maas-controller-775f5d4dfb-m6fdc\" (UID: \"a4eff9f4-e102-42b7-b33f-6f798dadb4e1\") " pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:29.683334 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.683267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:29.819002 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.818977 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-775f5d4dfb-m6fdc"] Jul 16 15:47:29.821155 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:47:29.821129 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4eff9f4_e102_42b7_b33f_6f798dadb4e1.slice/crio-e72b6ffb4c70f1f8093ca4e718bcee798e48e497a650432a41410a6797ab14b0 WatchSource:0}: Error finding container e72b6ffb4c70f1f8093ca4e718bcee798e48e497a650432a41410a6797ab14b0: Status 404 returned error can't find the container with id e72b6ffb4c70f1f8093ca4e718bcee798e48e497a650432a41410a6797ab14b0 Jul 16 15:47:29.822591 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:29.822570 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 15:47:30.780154 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:30.780113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" event={"ID":"a4eff9f4-e102-42b7-b33f-6f798dadb4e1","Type":"ContainerStarted","Data":"172aa3b30165c14c749b9f6d133435c2ccc696791c35ce52bce30897ff225209"} Jul 16 15:47:30.780548 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:30.780178 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" event={"ID":"a4eff9f4-e102-42b7-b33f-6f798dadb4e1","Type":"ContainerStarted","Data":"e72b6ffb4c70f1f8093ca4e718bcee798e48e497a650432a41410a6797ab14b0"} Jul 16 15:47:30.780548 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:30.780256 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:47:30.800729 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:30.800687 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" podStartSLOduration=1.147114485 podStartE2EDuration="1.800671987s" totalImagesPullingTime="653.557502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:47:30.798411924 +0000 UTC m=+705.236938746" watchObservedRunningTime="2026-07-16 15:47:30.800671987 +0000 UTC m=+705.239198838" Jul 16 15:47:41.794069 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:47:41.793987 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-775f5d4dfb-m6fdc" Jul 16 15:50:46.048437 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:50:46.048354 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:50:46.049044 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:50:46.048455 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:52:25.890242 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:25.890158 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f"] Jul 16 15:52:25.892898 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:25.890382 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" podUID="cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc" containerName="manager" containerID="cri-o://8b8c8697a5ef3473a2dd6fdaea467dcf73f604961dcfaa786b141867b9eb7352" gracePeriod=10 Jul 16 15:52:28.233144 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.233118 2573 generic.go:356] "Generic (PLEG): container finished" podID="cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc" containerID="8b8c8697a5ef3473a2dd6fdaea467dcf73f604961dcfaa786b141867b9eb7352" exitCode=0 Jul 16 15:52:28.233518 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.233188 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" event={"ID":"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc","Type":"ContainerDied","Data":"8b8c8697a5ef3473a2dd6fdaea467dcf73f604961dcfaa786b141867b9eb7352"} Jul 16 15:52:28.252918 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.252896 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:52:28.364311 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.364209 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-extensions-socket-volume\") pod \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\" (UID: \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\") " Jul 16 15:52:28.364311 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.364289 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmqjj\" (UniqueName: \"kubernetes.io/projected/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-kube-api-access-zmqjj\") pod \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\" (UID: \"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc\") " Jul 16 15:52:28.364602 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.364579 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc" (UID: "cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 15:52:28.366305 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.366278 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-kube-api-access-zmqjj" (OuterVolumeSpecName: "kube-api-access-zmqjj") pod "cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc" (UID: "cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc"). InnerVolumeSpecName "kube-api-access-zmqjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:52:28.465864 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.465822 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-extensions-socket-volume\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:52:28.465864 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:28.465865 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmqjj\" (UniqueName: \"kubernetes.io/projected/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc-kube-api-access-zmqjj\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:52:29.239457 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:29.239430 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" Jul 16 15:52:29.239941 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:29.239421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f" event={"ID":"cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc","Type":"ContainerDied","Data":"77b5de1d995a94911dd712118bcf7c53cee05e5a914673f0a1e52495e53f4442"} Jul 16 15:52:29.239941 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:29.239624 2573 scope.go:117] "RemoveContainer" containerID="8b8c8697a5ef3473a2dd6fdaea467dcf73f604961dcfaa786b141867b9eb7352" Jul 16 15:52:29.261375 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:29.261350 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f"] Jul 16 15:52:29.271072 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:29.271046 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s4z4f"] Jul 16 15:52:30.383750 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:52:30.383723 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc" path="/var/lib/kubelet/pods/cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc/volumes" Jul 16 15:53:27.122748 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.122713 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf"] Jul 16 15:53:27.123214 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.123171 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc" containerName="manager" Jul 16 15:53:27.123214 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.123184 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc" containerName="manager" Jul 16 15:53:27.123293 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.123261 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb7bfd0a-cf3f-4bdb-908e-f4a5e5b561cc" containerName="manager" Jul 16 15:53:27.126028 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.126009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:27.130542 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.130525 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nlqrf\"" Jul 16 15:53:27.143708 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.143686 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf"] Jul 16 15:53:27.189997 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.189970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/848104d7-23a4-404a-8447-1a09e78f714a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gqpkf\" (UID: \"848104d7-23a4-404a-8447-1a09e78f714a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:27.190129 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.190010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9nf\" (UniqueName: \"kubernetes.io/projected/848104d7-23a4-404a-8447-1a09e78f714a-kube-api-access-nk9nf\") pod \"kuadrant-operator-controller-manager-86878c769f-gqpkf\" (UID: \"848104d7-23a4-404a-8447-1a09e78f714a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:27.290586 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.290555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/848104d7-23a4-404a-8447-1a09e78f714a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gqpkf\" (UID: \"848104d7-23a4-404a-8447-1a09e78f714a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:27.290586 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.290589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9nf\" (UniqueName: \"kubernetes.io/projected/848104d7-23a4-404a-8447-1a09e78f714a-kube-api-access-nk9nf\") pod \"kuadrant-operator-controller-manager-86878c769f-gqpkf\" (UID: \"848104d7-23a4-404a-8447-1a09e78f714a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:27.290970 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.290949 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/848104d7-23a4-404a-8447-1a09e78f714a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gqpkf\" (UID: \"848104d7-23a4-404a-8447-1a09e78f714a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:27.308805 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.308784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9nf\" (UniqueName: \"kubernetes.io/projected/848104d7-23a4-404a-8447-1a09e78f714a-kube-api-access-nk9nf\") pod \"kuadrant-operator-controller-manager-86878c769f-gqpkf\" (UID: \"848104d7-23a4-404a-8447-1a09e78f714a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:27.436579 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.436504 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:27.571371 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:27.571333 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf"] Jul 16 15:53:27.574529 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:53:27.574507 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848104d7_23a4_404a_8447_1a09e78f714a.slice/crio-bbe7bc455de7d0dd1fd7aba961601a79280f8b45f28a9c4434e4b700dfb1cf28 WatchSource:0}: Error finding container bbe7bc455de7d0dd1fd7aba961601a79280f8b45f28a9c4434e4b700dfb1cf28: Status 404 returned error can't find the container with id bbe7bc455de7d0dd1fd7aba961601a79280f8b45f28a9c4434e4b700dfb1cf28 Jul 16 15:53:28.526041 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:28.526006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" event={"ID":"848104d7-23a4-404a-8447-1a09e78f714a","Type":"ContainerStarted","Data":"8346b95e1521ece5bcda0236f8c4bd2a650abf6feb6536a4c5f330ae903d0a02"} Jul 16 15:53:28.526579 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:28.526058 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" event={"ID":"848104d7-23a4-404a-8447-1a09e78f714a","Type":"ContainerStarted","Data":"bbe7bc455de7d0dd1fd7aba961601a79280f8b45f28a9c4434e4b700dfb1cf28"} Jul 16 15:53:28.526579 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:28.526087 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:53:28.551385 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:28.551341 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" podStartSLOduration=1.551325207 podStartE2EDuration="1.551325207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:53:28.546916185 +0000 UTC m=+1062.985443008" watchObservedRunningTime="2026-07-16 15:53:28.551325207 +0000 UTC m=+1062.989852030" Jul 16 15:53:39.532804 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:53:39.532771 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gqpkf" Jul 16 15:55:46.079214 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:55:46.079188 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:55:46.082216 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:55:46.082196 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 15:59:53.432803 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.432766 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc"] Jul 16 15:59:53.436416 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.436397 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.439139 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.439117 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9873c762-serving-cert\"" Jul 16 15:59:53.459704 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.459673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-openshift-service-ca\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.459825 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.459766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrb7\" (UniqueName: \"kubernetes.io/projected/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-kube-api-access-xjrb7\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.459931 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.459906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-maas-api-tls\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.464720 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.464699 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc"] Jul 16 15:59:53.561635 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.561574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-openshift-service-ca\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.561819 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.561715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrb7\" (UniqueName: \"kubernetes.io/projected/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-kube-api-access-xjrb7\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.561819 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.561768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-maas-api-tls\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.562275 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.562252 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-openshift-service-ca\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.564184 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.564162 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-maas-api-tls\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.572862 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.572837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrb7\" (UniqueName: \"kubernetes.io/projected/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-kube-api-access-xjrb7\") pod \"maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.746665 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.746551 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:53.898274 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.898251 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc"] Jul 16 15:59:53.900087 ip-10-0-128-48 kubenswrapper[2573]: W0716 15:59:53.900061 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d9febd_f1f9_41f2_98ab_9fcdbb5c46ff.slice/crio-688667c495c9342a8441db74560d6eec744bcbb61edb19f1672e8851ebe7c76b WatchSource:0}: Error finding container 688667c495c9342a8441db74560d6eec744bcbb61edb19f1672e8851ebe7c76b: Status 404 returned error can't find the container with id 688667c495c9342a8441db74560d6eec744bcbb61edb19f1672e8851ebe7c76b Jul 16 15:59:53.901500 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:53.901478 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 15:59:54.438886 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:54.438846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" event={"ID":"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff","Type":"ContainerStarted","Data":"688667c495c9342a8441db74560d6eec744bcbb61edb19f1672e8851ebe7c76b"} Jul 16 15:59:55.445219 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:55.445183 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" event={"ID":"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff","Type":"ContainerStarted","Data":"d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3"} Jul 16 15:59:55.445764 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:55.445334 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:55.488599 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:55.488549 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" podStartSLOduration=1.831340737 podStartE2EDuration="2.488534547s" totalImagesPullingTime="657.19381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 15:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 15:59:55.486716082 +0000 UTC m=+1449.925242906" watchObservedRunningTime="2026-07-16 15:59:55.488534547 +0000 UTC m=+1449.927061369" Jul 16 15:59:58.812294 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:58.812259 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc"] Jul 16 15:59:58.812778 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:58.812515 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" podUID="c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" containerName="maas-api" containerID="cri-o://d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3" gracePeriod=30 Jul 16 15:59:59.053179 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.053151 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:59.115727 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.115663 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-openshift-service-ca\") pod \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " Jul 16 15:59:59.115850 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.115736 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-maas-api-tls\") pod \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " Jul 16 15:59:59.115850 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.115830 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjrb7\" (UniqueName: \"kubernetes.io/projected/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-kube-api-access-xjrb7\") pod \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\" (UID: \"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff\") " Jul 16 15:59:59.115997 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.115976 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" (UID: "c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 15:59:59.116117 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.116098 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:59:59.117799 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.117777 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" (UID: "c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 15:59:59.117868 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.117813 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-kube-api-access-xjrb7" (OuterVolumeSpecName: "kube-api-access-xjrb7") pod "c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" (UID: "c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff"). InnerVolumeSpecName "kube-api-access-xjrb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 15:59:59.216986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.216951 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrb7\" (UniqueName: \"kubernetes.io/projected/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-kube-api-access-xjrb7\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:59:59.216986 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.216983 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 15:59:59.466787 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.466695 2573 generic.go:356] "Generic (PLEG): container finished" podID="c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" containerID="d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3" exitCode=0 Jul 16 15:59:59.466787 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.466770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" event={"ID":"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff","Type":"ContainerDied","Data":"d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3"} Jul 16 15:59:59.467054 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.467026 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" event={"ID":"c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff","Type":"ContainerDied","Data":"688667c495c9342a8441db74560d6eec744bcbb61edb19f1672e8851ebe7c76b"} Jul 16 15:59:59.467124 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.466736 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc" Jul 16 15:59:59.467194 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.467174 2573 scope.go:117] "RemoveContainer" containerID="d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3" Jul 16 15:59:59.482603 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.482589 2573 scope.go:117] "RemoveContainer" containerID="d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3" Jul 16 15:59:59.482867 ip-10-0-128-48 kubenswrapper[2573]: E0716 15:59:59.482850 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3\": container with ID starting with d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3 not found: ID does not exist" containerID="d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3" Jul 16 15:59:59.482920 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.482875 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3"} err="failed to get container status \"d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3\": rpc error: code = NotFound desc = could not find container \"d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3\": container with ID starting with d54a449973c68b95d2b8586c71c30e1690d5937f19970bdb2e0172f825103bf3 not found: ID does not exist" Jul 16 15:59:59.502528 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.502505 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc"] Jul 16 15:59:59.508191 ip-10-0-128-48 kubenswrapper[2573]: I0716 15:59:59.508171 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9873c762-77dd855cc7-b2lqc"] Jul 16 16:00:00.136854 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.136818 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht"] Jul 16 16:00:00.137282 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.137269 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" containerName="maas-api" Jul 16 16:00:00.137331 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.137284 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" containerName="maas-api" Jul 16 16:00:00.137371 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.137362 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" containerName="maas-api" Jul 16 16:00:00.142209 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.142191 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" Jul 16 16:00:00.152046 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.152021 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht"] Jul 16 16:00:00.224969 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.224940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rm4\" (UniqueName: \"kubernetes.io/projected/34a710e3-a1ae-4d1c-a651-f1ed5a67f63b-kube-api-access-52rm4\") pod \"maas-api-key-cleanup-29736960-fq5ht\" (UID: \"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" Jul 16 16:00:00.325810 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.325776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rm4\" (UniqueName: \"kubernetes.io/projected/34a710e3-a1ae-4d1c-a651-f1ed5a67f63b-kube-api-access-52rm4\") pod \"maas-api-key-cleanup-29736960-fq5ht\" (UID: \"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" Jul 16 16:00:00.336142 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.336110 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rm4\" (UniqueName: \"kubernetes.io/projected/34a710e3-a1ae-4d1c-a651-f1ed5a67f63b-kube-api-access-52rm4\") pod \"maas-api-key-cleanup-29736960-fq5ht\" (UID: \"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" Jul 16 16:00:00.384079 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.384050 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff" path="/var/lib/kubelet/pods/c2d9febd-f1f9-41f2-98ab-9fcdbb5c46ff/volumes" Jul 16 16:00:00.453603 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.453518 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" Jul 16 16:00:00.591565 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:00.591542 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht"] Jul 16 16:00:00.593431 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:00:00.593407 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a710e3_a1ae_4d1c_a651_f1ed5a67f63b.slice/crio-453e7367932462f4d229f08a150769e639f243cf58f611693329957a31d00987 WatchSource:0}: Error finding container 453e7367932462f4d229f08a150769e639f243cf58f611693329957a31d00987: Status 404 returned error can't find the container with id 453e7367932462f4d229f08a150769e639f243cf58f611693329957a31d00987 Jul 16 16:00:01.481298 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:01.481258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" event={"ID":"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b","Type":"ContainerStarted","Data":"89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3"} Jul 16 16:00:01.481768 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:01.481323 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" event={"ID":"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b","Type":"ContainerStarted","Data":"453e7367932462f4d229f08a150769e639f243cf58f611693329957a31d00987"} Jul 16 16:00:01.501014 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:01.500970 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" podStartSLOduration=1.500953742 podStartE2EDuration="1.500953742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:00:01.498687422 +0000 UTC m=+1455.937214247" watchObservedRunningTime="2026-07-16 16:00:01.500953742 +0000 UTC m=+1455.939480564" Jul 16 16:00:06.808998 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.808965 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc"] Jul 16 16:00:06.812752 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.812735 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:06.816382 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.816366 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9717f3de-serving-cert\"" Jul 16 16:00:06.826315 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.826290 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc"] Jul 16 16:00:06.888191 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.888160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774f57ae-8c04-4d06-a27f-725789d19474-openshift-service-ca\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:06.888335 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.888205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rtt\" (UniqueName: \"kubernetes.io/projected/774f57ae-8c04-4d06-a27f-725789d19474-kube-api-access-l2rtt\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:06.888335 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.888231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:06.989305 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.989272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774f57ae-8c04-4d06-a27f-725789d19474-openshift-service-ca\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:06.989481 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.989311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rtt\" (UniqueName: \"kubernetes.io/projected/774f57ae-8c04-4d06-a27f-725789d19474-kube-api-access-l2rtt\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:06.989481 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.989333 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:06.989481 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:00:06.989463 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-serving-cert: secret "maas-api-e2e-ait-9717f3de-serving-cert" not found Jul 16 16:00:06.989698 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:00:06.989521 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls podName:774f57ae-8c04-4d06-a27f-725789d19474 nodeName:}" failed. No retries permitted until 2026-07-16 16:00:07.48950494 +0000 UTC m=+1461.928031746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls") pod "maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" (UID: "774f57ae-8c04-4d06-a27f-725789d19474") : secret "maas-api-e2e-ait-9717f3de-serving-cert" not found Jul 16 16:00:06.990012 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:06.989990 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774f57ae-8c04-4d06-a27f-725789d19474-openshift-service-ca\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:07.005702 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:07.005679 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rtt\" (UniqueName: \"kubernetes.io/projected/774f57ae-8c04-4d06-a27f-725789d19474-kube-api-access-l2rtt\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:07.493576 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:07.493542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:07.495809 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:07.495785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls\") pod \"maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:07.723110 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:07.723070 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:07.860907 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:07.860877 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc"] Jul 16 16:00:07.863671 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:00:07.863644 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774f57ae_8c04_4d06_a27f_725789d19474.slice/crio-aba7d177c42da15abf5c7bb74a2d23c03a2709b5353d01b64fdd1cfce13d40c3 WatchSource:0}: Error finding container aba7d177c42da15abf5c7bb74a2d23c03a2709b5353d01b64fdd1cfce13d40c3: Status 404 returned error can't find the container with id aba7d177c42da15abf5c7bb74a2d23c03a2709b5353d01b64fdd1cfce13d40c3 Jul 16 16:00:08.518017 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:08.517982 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" event={"ID":"774f57ae-8c04-4d06-a27f-725789d19474","Type":"ContainerStarted","Data":"78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85"} Jul 16 16:00:08.518188 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:08.518034 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" event={"ID":"774f57ae-8c04-4d06-a27f-725789d19474","Type":"ContainerStarted","Data":"aba7d177c42da15abf5c7bb74a2d23c03a2709b5353d01b64fdd1cfce13d40c3"} Jul 16 16:00:08.518188 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:08.518079 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:08.549485 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:08.549435 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" podStartSLOduration=2.065790301 podStartE2EDuration="2.549420094s" totalImagesPullingTime="483.629793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:00:08.547351652 +0000 UTC m=+1462.985878475" watchObservedRunningTime="2026-07-16 16:00:08.549420094 +0000 UTC m=+1462.987946917" Jul 16 16:00:12.591240 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.591202 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc"] Jul 16 16:00:12.591771 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.591481 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" podUID="774f57ae-8c04-4d06-a27f-725789d19474" containerName="maas-api" containerID="cri-o://78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85" gracePeriod=30 Jul 16 16:00:12.839268 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.839242 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:12.942183 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.942116 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls\") pod \"774f57ae-8c04-4d06-a27f-725789d19474\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " Jul 16 16:00:12.942183 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.942168 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774f57ae-8c04-4d06-a27f-725789d19474-openshift-service-ca\") pod \"774f57ae-8c04-4d06-a27f-725789d19474\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " Jul 16 16:00:12.942346 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.942211 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2rtt\" (UniqueName: \"kubernetes.io/projected/774f57ae-8c04-4d06-a27f-725789d19474-kube-api-access-l2rtt\") pod \"774f57ae-8c04-4d06-a27f-725789d19474\" (UID: \"774f57ae-8c04-4d06-a27f-725789d19474\") " Jul 16 16:00:12.942539 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.942515 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774f57ae-8c04-4d06-a27f-725789d19474-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "774f57ae-8c04-4d06-a27f-725789d19474" (UID: "774f57ae-8c04-4d06-a27f-725789d19474"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:00:12.944180 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.944159 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "774f57ae-8c04-4d06-a27f-725789d19474" (UID: "774f57ae-8c04-4d06-a27f-725789d19474"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:00:12.944246 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:12.944221 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774f57ae-8c04-4d06-a27f-725789d19474-kube-api-access-l2rtt" (OuterVolumeSpecName: "kube-api-access-l2rtt") pod "774f57ae-8c04-4d06-a27f-725789d19474" (UID: "774f57ae-8c04-4d06-a27f-725789d19474"). InnerVolumeSpecName "kube-api-access-l2rtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:00:13.043027 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.042996 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f57ae-8c04-4d06-a27f-725789d19474-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:00:13.043027 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.043024 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774f57ae-8c04-4d06-a27f-725789d19474-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:00:13.043027 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.043034 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2rtt\" (UniqueName: \"kubernetes.io/projected/774f57ae-8c04-4d06-a27f-725789d19474-kube-api-access-l2rtt\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:00:13.543980 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.543946 2573 generic.go:356] "Generic (PLEG): container finished" podID="774f57ae-8c04-4d06-a27f-725789d19474" containerID="78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85" exitCode=0 Jul 16 16:00:13.544164 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.543994 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" Jul 16 16:00:13.544164 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.544035 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" event={"ID":"774f57ae-8c04-4d06-a27f-725789d19474","Type":"ContainerDied","Data":"78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85"} Jul 16 16:00:13.544164 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.544101 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc" event={"ID":"774f57ae-8c04-4d06-a27f-725789d19474","Type":"ContainerDied","Data":"aba7d177c42da15abf5c7bb74a2d23c03a2709b5353d01b64fdd1cfce13d40c3"} Jul 16 16:00:13.544164 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.544132 2573 scope.go:117] "RemoveContainer" containerID="78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85" Jul 16 16:00:13.552873 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.552852 2573 scope.go:117] "RemoveContainer" containerID="78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85" Jul 16 16:00:13.553119 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:00:13.553100 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85\": container with ID starting with 78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85 not found: ID does not exist" containerID="78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85" Jul 16 16:00:13.553183 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.553130 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85"} err="failed to get container status \"78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85\": rpc error: code = NotFound desc = could not find container \"78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85\": container with ID starting with 78e4fb06ae5330247e24230d6917c0b5d379f7982036eeffda858059363e1c85 not found: ID does not exist" Jul 16 16:00:13.573632 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.573598 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc"] Jul 16 16:00:13.584262 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:13.584240 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9717f3de-5c4c7f58b-hpsnc"] Jul 16 16:00:14.383473 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:14.383442 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="774f57ae-8c04-4d06-a27f-725789d19474" path="/var/lib/kubelet/pods/774f57ae-8c04-4d06-a27f-725789d19474/volumes" Jul 16 16:00:26.726561 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.726528 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr"] Jul 16 16:00:26.726979 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.726931 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="774f57ae-8c04-4d06-a27f-725789d19474" containerName="maas-api" Jul 16 16:00:26.726979 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.726944 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="774f57ae-8c04-4d06-a27f-725789d19474" containerName="maas-api" Jul 16 16:00:26.727051 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.727016 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="774f57ae-8c04-4d06-a27f-725789d19474" containerName="maas-api" Jul 16 16:00:26.730202 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.730185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:26.736450 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.736429 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e590f3b0-serving-cert\"" Jul 16 16:00:26.766324 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.766299 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr"] Jul 16 16:00:26.868465 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.868431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d13552-c23e-4df5-9919-55689ff9edde-openshift-service-ca\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:26.868465 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.868464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl8bj\" (UniqueName: \"kubernetes.io/projected/40d13552-c23e-4df5-9919-55689ff9edde-kube-api-access-pl8bj\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:26.868713 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.868639 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:26.969095 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.969056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:26.969247 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.969119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d13552-c23e-4df5-9919-55689ff9edde-openshift-service-ca\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:26.969247 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.969145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8bj\" (UniqueName: \"kubernetes.io/projected/40d13552-c23e-4df5-9919-55689ff9edde-kube-api-access-pl8bj\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:26.969247 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:00:26.969200 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-serving-cert: secret "maas-api-e2e-derive-e590f3b0-serving-cert" not found Jul 16 16:00:26.969410 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:00:26.969275 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls podName:40d13552-c23e-4df5-9919-55689ff9edde nodeName:}" failed. No retries permitted until 2026-07-16 16:00:27.469258536 +0000 UTC m=+1481.907785341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls") pod "maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" (UID: "40d13552-c23e-4df5-9919-55689ff9edde") : secret "maas-api-e2e-derive-e590f3b0-serving-cert" not found Jul 16 16:00:26.969789 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.969765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d13552-c23e-4df5-9919-55689ff9edde-openshift-service-ca\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:26.983815 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:26.983751 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8bj\" (UniqueName: \"kubernetes.io/projected/40d13552-c23e-4df5-9919-55689ff9edde-kube-api-access-pl8bj\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:27.473673 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:27.473636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:27.475959 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:27.475931 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls\") pod \"maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:27.641223 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:27.641192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:27.784523 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:27.784485 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr"] Jul 16 16:00:27.788309 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:00:27.788280 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d13552_c23e_4df5_9919_55689ff9edde.slice/crio-b7d013ed0ae571f95eedb4645b3014cd1cbd65fcd85deabfaf8b9ed76928c612 WatchSource:0}: Error finding container b7d013ed0ae571f95eedb4645b3014cd1cbd65fcd85deabfaf8b9ed76928c612: Status 404 returned error can't find the container with id b7d013ed0ae571f95eedb4645b3014cd1cbd65fcd85deabfaf8b9ed76928c612 Jul 16 16:00:28.623871 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:28.623834 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" event={"ID":"40d13552-c23e-4df5-9919-55689ff9edde","Type":"ContainerStarted","Data":"37692f9987ee92d1cf2da21e358a0465ccd89a5eed3fafded119cc886258e586"} Jul 16 16:00:28.624026 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:28.623915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" event={"ID":"40d13552-c23e-4df5-9919-55689ff9edde","Type":"ContainerStarted","Data":"b7d013ed0ae571f95eedb4645b3014cd1cbd65fcd85deabfaf8b9ed76928c612"} Jul 16 16:00:28.624026 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:28.623979 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:28.646100 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:28.646057 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" podStartSLOduration=1.993416681 podStartE2EDuration="2.646042853s" totalImagesPullingTime="652.626172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:00:28.644435506 +0000 UTC m=+1483.082962329" watchObservedRunningTime="2026-07-16 16:00:28.646042853 +0000 UTC m=+1483.084569677" Jul 16 16:00:31.395148 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.395118 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr"] Jul 16 16:00:31.395513 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.395287 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" podUID="40d13552-c23e-4df5-9919-55689ff9edde" containerName="maas-api" containerID="cri-o://37692f9987ee92d1cf2da21e358a0465ccd89a5eed3fafded119cc886258e586" gracePeriod=30 Jul 16 16:00:31.640879 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.640850 2573 generic.go:356] "Generic (PLEG): container finished" podID="40d13552-c23e-4df5-9919-55689ff9edde" containerID="37692f9987ee92d1cf2da21e358a0465ccd89a5eed3fafded119cc886258e586" exitCode=0 Jul 16 16:00:31.641018 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.640923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" event={"ID":"40d13552-c23e-4df5-9919-55689ff9edde","Type":"ContainerDied","Data":"37692f9987ee92d1cf2da21e358a0465ccd89a5eed3fafded119cc886258e586"} Jul 16 16:00:31.641018 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.640971 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" event={"ID":"40d13552-c23e-4df5-9919-55689ff9edde","Type":"ContainerDied","Data":"b7d013ed0ae571f95eedb4645b3014cd1cbd65fcd85deabfaf8b9ed76928c612"} Jul 16 16:00:31.641018 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.640988 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7d013ed0ae571f95eedb4645b3014cd1cbd65fcd85deabfaf8b9ed76928c612" Jul 16 16:00:31.648821 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.648766 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:31.717487 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.717459 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d13552-c23e-4df5-9919-55689ff9edde-openshift-service-ca\") pod \"40d13552-c23e-4df5-9919-55689ff9edde\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " Jul 16 16:00:31.717662 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.717503 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl8bj\" (UniqueName: \"kubernetes.io/projected/40d13552-c23e-4df5-9919-55689ff9edde-kube-api-access-pl8bj\") pod \"40d13552-c23e-4df5-9919-55689ff9edde\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " Jul 16 16:00:31.717662 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.717526 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls\") pod \"40d13552-c23e-4df5-9919-55689ff9edde\" (UID: \"40d13552-c23e-4df5-9919-55689ff9edde\") " Jul 16 16:00:31.717844 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.717820 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d13552-c23e-4df5-9919-55689ff9edde-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40d13552-c23e-4df5-9919-55689ff9edde" (UID: "40d13552-c23e-4df5-9919-55689ff9edde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:00:31.719659 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.719602 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40d13552-c23e-4df5-9919-55689ff9edde" (UID: "40d13552-c23e-4df5-9919-55689ff9edde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:00:31.719786 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.719734 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d13552-c23e-4df5-9919-55689ff9edde-kube-api-access-pl8bj" (OuterVolumeSpecName: "kube-api-access-pl8bj") pod "40d13552-c23e-4df5-9919-55689ff9edde" (UID: "40d13552-c23e-4df5-9919-55689ff9edde"). InnerVolumeSpecName "kube-api-access-pl8bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:00:31.819035 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.819000 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d13552-c23e-4df5-9919-55689ff9edde-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:00:31.819035 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.819029 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl8bj\" (UniqueName: \"kubernetes.io/projected/40d13552-c23e-4df5-9919-55689ff9edde-kube-api-access-pl8bj\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:00:31.819223 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:31.819044 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d13552-c23e-4df5-9919-55689ff9edde-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:00:32.645887 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:32.645809 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr" Jul 16 16:00:32.676914 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:32.676883 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr"] Jul 16 16:00:32.684669 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:32.684644 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e590f3b0-78d58c8667-w84rr"] Jul 16 16:00:34.384064 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:34.384033 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d13552-c23e-4df5-9919-55689ff9edde" path="/var/lib/kubelet/pods/40d13552-c23e-4df5-9919-55689ff9edde/volumes" Jul 16 16:00:46.110257 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:46.110229 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 16:00:46.114647 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:00:46.114608 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 16:01:00.012275 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:00.012241 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht"] Jul 16 16:01:00.012849 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:00.012449 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" podUID="34a710e3-a1ae-4d1c-a651-f1ed5a67f63b" containerName="cleanup" containerID="cri-o://89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3" gracePeriod=30 Jul 16 16:01:30.191913 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.191885 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" Jul 16 16:01:30.249988 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.249959 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rm4\" (UniqueName: \"kubernetes.io/projected/34a710e3-a1ae-4d1c-a651-f1ed5a67f63b-kube-api-access-52rm4\") pod \"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b\" (UID: \"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b\") " Jul 16 16:01:30.251896 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.251871 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a710e3-a1ae-4d1c-a651-f1ed5a67f63b-kube-api-access-52rm4" (OuterVolumeSpecName: "kube-api-access-52rm4") pod "34a710e3-a1ae-4d1c-a651-f1ed5a67f63b" (UID: "34a710e3-a1ae-4d1c-a651-f1ed5a67f63b"). InnerVolumeSpecName "kube-api-access-52rm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:01:30.351563 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.351496 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rm4\" (UniqueName: \"kubernetes.io/projected/34a710e3-a1ae-4d1c-a651-f1ed5a67f63b-kube-api-access-52rm4\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:01:30.938631 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.938579 2573 generic.go:356] "Generic (PLEG): container finished" podID="34a710e3-a1ae-4d1c-a651-f1ed5a67f63b" containerID="89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3" exitCode=137 Jul 16 16:01:30.938918 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.938642 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" Jul 16 16:01:30.938918 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.938648 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" event={"ID":"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b","Type":"ContainerDied","Data":"89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3"} Jul 16 16:01:30.938918 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.938703 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht" event={"ID":"34a710e3-a1ae-4d1c-a651-f1ed5a67f63b","Type":"ContainerDied","Data":"453e7367932462f4d229f08a150769e639f243cf58f611693329957a31d00987"} Jul 16 16:01:30.938918 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.938731 2573 scope.go:117] "RemoveContainer" containerID="89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3" Jul 16 16:01:30.950096 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.950073 2573 scope.go:117] "RemoveContainer" containerID="89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3" Jul 16 16:01:30.950407 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:01:30.950384 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3\": container with ID starting with 89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3 not found: ID does not exist" containerID="89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3" Jul 16 16:01:30.950486 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.950418 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3"} err="failed to get container status \"89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3\": rpc error: code = NotFound desc = could not find container \"89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3\": container with ID starting with 89327cc1e0e89a0d8a455f41fb9ba4911f914aaf8c6c32e344a9152ab0cb3fa3 not found: ID does not exist" Jul 16 16:01:30.969575 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.969554 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht"] Jul 16 16:01:30.978210 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:30.978191 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736960-fq5ht"] Jul 16 16:01:32.383674 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:01:32.383644 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34a710e3-a1ae-4d1c-a651-f1ed5a67f63b" path="/var/lib/kubelet/pods/34a710e3-a1ae-4d1c-a651-f1ed5a67f63b/volumes" Jul 16 16:02:29.580928 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.580896 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl"] Jul 16 16:02:29.581332 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.581279 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a710e3-a1ae-4d1c-a651-f1ed5a67f63b" containerName="cleanup" Jul 16 16:02:29.581332 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.581290 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a710e3-a1ae-4d1c-a651-f1ed5a67f63b" containerName="cleanup" Jul 16 16:02:29.581332 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.581318 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d13552-c23e-4df5-9919-55689ff9edde" containerName="maas-api" Jul 16 16:02:29.581332 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.581323 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d13552-c23e-4df5-9919-55689ff9edde" containerName="maas-api" Jul 16 16:02:29.581459 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.581382 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a710e3-a1ae-4d1c-a651-f1ed5a67f63b" containerName="cleanup" Jul 16 16:02:29.581459 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.581390 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d13552-c23e-4df5-9919-55689ff9edde" containerName="maas-api" Jul 16 16:02:29.584272 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.584255 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:29.588312 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.588284 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3774490c-serving-cert\"" Jul 16 16:02:29.646264 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.646234 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl"] Jul 16 16:02:29.680545 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.680516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:29.680712 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.680547 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nnv\" (UniqueName: \"kubernetes.io/projected/a8d9b910-1477-4317-9bab-c037f50aa025-kube-api-access-q2nnv\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:29.680712 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.680692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d9b910-1477-4317-9bab-c037f50aa025-openshift-service-ca\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:29.781510 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.781482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:29.781689 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.781516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nnv\" (UniqueName: \"kubernetes.io/projected/a8d9b910-1477-4317-9bab-c037f50aa025-kube-api-access-q2nnv\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:29.781689 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.781591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d9b910-1477-4317-9bab-c037f50aa025-openshift-service-ca\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:29.781689 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:02:29.781670 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-serving-cert: secret "maas-api-e2e-mt-3774490c-serving-cert" not found Jul 16 16:02:29.781804 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:02:29.781766 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls podName:a8d9b910-1477-4317-9bab-c037f50aa025 nodeName:}" failed. No retries permitted until 2026-07-16 16:02:30.281746595 +0000 UTC m=+1604.720273396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls") pod "maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" (UID: "a8d9b910-1477-4317-9bab-c037f50aa025") : secret "maas-api-e2e-mt-3774490c-serving-cert" not found Jul 16 16:02:29.782185 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.782166 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d9b910-1477-4317-9bab-c037f50aa025-openshift-service-ca\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:29.808818 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:29.808794 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nnv\" (UniqueName: \"kubernetes.io/projected/a8d9b910-1477-4317-9bab-c037f50aa025-kube-api-access-q2nnv\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:30.287597 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:30.287563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:30.290083 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:30.290051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls\") pod \"maas-api-e2e-mt-3774490c-79cbf98f94-9btvl\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:30.494587 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:30.494552 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:30.655528 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:30.655440 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl"] Jul 16 16:02:30.657849 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:02:30.657823 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d9b910_1477_4317_9bab_c037f50aa025.slice/crio-c450f94e70f6af01a7182ba4a0dc52dd5a6e7ba6457de8e8ea5da320dc288426 WatchSource:0}: Error finding container c450f94e70f6af01a7182ba4a0dc52dd5a6e7ba6457de8e8ea5da320dc288426: Status 404 returned error can't find the container with id c450f94e70f6af01a7182ba4a0dc52dd5a6e7ba6457de8e8ea5da320dc288426 Jul 16 16:02:31.226905 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:31.226853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" event={"ID":"a8d9b910-1477-4317-9bab-c037f50aa025","Type":"ContainerStarted","Data":"e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c"} Jul 16 16:02:31.227069 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:31.226959 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:31.227069 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:31.226984 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" event={"ID":"a8d9b910-1477-4317-9bab-c037f50aa025","Type":"ContainerStarted","Data":"c450f94e70f6af01a7182ba4a0dc52dd5a6e7ba6457de8e8ea5da320dc288426"} Jul 16 16:02:37.237806 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:37.237773 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:37.279663 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:37.279599 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" podStartSLOduration=7.859101467 podStartE2EDuration="8.279583262s" totalImagesPullingTime="420.481795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:02:31.281704582 +0000 UTC m=+1605.720231407" watchObservedRunningTime="2026-07-16 16:02:37.279583262 +0000 UTC m=+1611.718110087" Jul 16 16:02:44.215866 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.215788 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl"] Jul 16 16:02:44.216248 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.215963 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" podUID="a8d9b910-1477-4317-9bab-c037f50aa025" containerName="maas-api" containerID="cri-o://e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c" gracePeriod=30 Jul 16 16:02:44.461369 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.461342 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:44.525441 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.525402 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2nnv\" (UniqueName: \"kubernetes.io/projected/a8d9b910-1477-4317-9bab-c037f50aa025-kube-api-access-q2nnv\") pod \"a8d9b910-1477-4317-9bab-c037f50aa025\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " Jul 16 16:02:44.525441 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.525437 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls\") pod \"a8d9b910-1477-4317-9bab-c037f50aa025\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " Jul 16 16:02:44.525728 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.525455 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d9b910-1477-4317-9bab-c037f50aa025-openshift-service-ca\") pod \"a8d9b910-1477-4317-9bab-c037f50aa025\" (UID: \"a8d9b910-1477-4317-9bab-c037f50aa025\") " Jul 16 16:02:44.525855 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.525833 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d9b910-1477-4317-9bab-c037f50aa025-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8d9b910-1477-4317-9bab-c037f50aa025" (UID: "a8d9b910-1477-4317-9bab-c037f50aa025"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:02:44.527470 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.527432 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8d9b910-1477-4317-9bab-c037f50aa025" (UID: "a8d9b910-1477-4317-9bab-c037f50aa025"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:02:44.527571 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.527505 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d9b910-1477-4317-9bab-c037f50aa025-kube-api-access-q2nnv" (OuterVolumeSpecName: "kube-api-access-q2nnv") pod "a8d9b910-1477-4317-9bab-c037f50aa025" (UID: "a8d9b910-1477-4317-9bab-c037f50aa025"). InnerVolumeSpecName "kube-api-access-q2nnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:02:44.626249 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.626204 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nnv\" (UniqueName: \"kubernetes.io/projected/a8d9b910-1477-4317-9bab-c037f50aa025-kube-api-access-q2nnv\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:02:44.626249 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.626247 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d9b910-1477-4317-9bab-c037f50aa025-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:02:44.626466 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:44.626261 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d9b910-1477-4317-9bab-c037f50aa025-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:02:45.300987 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.300948 2573 generic.go:356] "Generic (PLEG): container finished" podID="a8d9b910-1477-4317-9bab-c037f50aa025" containerID="e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c" exitCode=0 Jul 16 16:02:45.301464 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.301039 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" event={"ID":"a8d9b910-1477-4317-9bab-c037f50aa025","Type":"ContainerDied","Data":"e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c"} Jul 16 16:02:45.301464 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.301113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" event={"ID":"a8d9b910-1477-4317-9bab-c037f50aa025","Type":"ContainerDied","Data":"c450f94e70f6af01a7182ba4a0dc52dd5a6e7ba6457de8e8ea5da320dc288426"} Jul 16 16:02:45.301464 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.301058 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl" Jul 16 16:02:45.301464 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.301148 2573 scope.go:117] "RemoveContainer" containerID="e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c" Jul 16 16:02:45.310244 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.310228 2573 scope.go:117] "RemoveContainer" containerID="e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c" Jul 16 16:02:45.310493 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:02:45.310474 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c\": container with ID starting with e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c not found: ID does not exist" containerID="e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c" Jul 16 16:02:45.310571 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.310507 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c"} err="failed to get container status \"e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c\": rpc error: code = NotFound desc = could not find container \"e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c\": container with ID starting with e27064fbfb804196095bae5dec685ab85810d5f49c7ae9f3a002996009a99b5c not found: ID does not exist" Jul 16 16:02:45.344762 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.344736 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl"] Jul 16 16:02:45.355923 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:45.355900 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3774490c-79cbf98f94-9btvl"] Jul 16 16:02:46.384173 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:02:46.384136 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d9b910-1477-4317-9bab-c037f50aa025" path="/var/lib/kubelet/pods/a8d9b910-1477-4317-9bab-c037f50aa025/volumes" Jul 16 16:03:25.465308 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.465268 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m"] Jul 16 16:03:25.465923 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.465901 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d9b910-1477-4317-9bab-c037f50aa025" containerName="maas-api" Jul 16 16:03:25.465923 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.465922 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d9b910-1477-4317-9bab-c037f50aa025" containerName="maas-api" Jul 16 16:03:25.466041 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.466029 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d9b910-1477-4317-9bab-c037f50aa025" containerName="maas-api" Jul 16 16:03:25.469212 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.469191 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.472809 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.472779 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ef9fd84b-serving-cert\"" Jul 16 16:03:25.495362 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.495343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6455c7f-5be7-447a-816e-ff0c9b597d65-openshift-service-ca\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.495450 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.495373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6455c7f-5be7-447a-816e-ff0c9b597d65-maas-api-tls\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.495450 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.495392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mww6d\" (UniqueName: \"kubernetes.io/projected/e6455c7f-5be7-447a-816e-ff0c9b597d65-kube-api-access-mww6d\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.520869 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.520846 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m"] Jul 16 16:03:25.596948 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.596914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6455c7f-5be7-447a-816e-ff0c9b597d65-openshift-service-ca\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.597134 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.596959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6455c7f-5be7-447a-816e-ff0c9b597d65-maas-api-tls\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.597134 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.596989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mww6d\" (UniqueName: \"kubernetes.io/projected/e6455c7f-5be7-447a-816e-ff0c9b597d65-kube-api-access-mww6d\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.597643 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.597579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6455c7f-5be7-447a-816e-ff0c9b597d65-openshift-service-ca\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.599464 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.599438 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6455c7f-5be7-447a-816e-ff0c9b597d65-maas-api-tls\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.622320 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.622294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww6d\" (UniqueName: \"kubernetes.io/projected/e6455c7f-5be7-447a-816e-ff0c9b597d65-kube-api-access-mww6d\") pod \"maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.779187 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.779155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:25.929477 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:25.929451 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m"] Jul 16 16:03:25.931121 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:03:25.931093 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6455c7f_5be7_447a_816e_ff0c9b597d65.slice/crio-2cfeeca6a94caefb684846fdff97750a6f25a8d1903a11cefd0d26787a466256 WatchSource:0}: Error finding container 2cfeeca6a94caefb684846fdff97750a6f25a8d1903a11cefd0d26787a466256: Status 404 returned error can't find the container with id 2cfeeca6a94caefb684846fdff97750a6f25a8d1903a11cefd0d26787a466256 Jul 16 16:03:26.508772 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:26.508738 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" event={"ID":"e6455c7f-5be7-447a-816e-ff0c9b597d65","Type":"ContainerStarted","Data":"2cfeeca6a94caefb684846fdff97750a6f25a8d1903a11cefd0d26787a466256"} Jul 16 16:03:27.514742 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:27.514708 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" event={"ID":"e6455c7f-5be7-447a-816e-ff0c9b597d65","Type":"ContainerStarted","Data":"53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15"} Jul 16 16:03:27.515131 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:27.514769 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:03:27.555669 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:27.555594 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" podStartSLOduration=1.7864250689999999 podStartE2EDuration="2.555575666s" totalImagesPullingTime="769.150597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:03:27.553948838 +0000 UTC m=+1661.992475662" watchObservedRunningTime="2026-07-16 16:03:27.555575666 +0000 UTC m=+1661.994102490" Jul 16 16:03:33.525147 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:03:33.525123 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:04:18.792470 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:18.792386 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m"] Jul 16 16:04:18.792989 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:18.792655 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" podUID="e6455c7f-5be7-447a-816e-ff0c9b597d65" containerName="maas-api" containerID="cri-o://53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15" gracePeriod=30 Jul 16 16:04:19.037867 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.037842 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:04:19.103989 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.103906 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mww6d\" (UniqueName: \"kubernetes.io/projected/e6455c7f-5be7-447a-816e-ff0c9b597d65-kube-api-access-mww6d\") pod \"e6455c7f-5be7-447a-816e-ff0c9b597d65\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " Jul 16 16:04:19.103989 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.103963 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6455c7f-5be7-447a-816e-ff0c9b597d65-openshift-service-ca\") pod \"e6455c7f-5be7-447a-816e-ff0c9b597d65\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " Jul 16 16:04:19.104214 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.104023 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6455c7f-5be7-447a-816e-ff0c9b597d65-maas-api-tls\") pod \"e6455c7f-5be7-447a-816e-ff0c9b597d65\" (UID: \"e6455c7f-5be7-447a-816e-ff0c9b597d65\") " Jul 16 16:04:19.104372 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.104339 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6455c7f-5be7-447a-816e-ff0c9b597d65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6455c7f-5be7-447a-816e-ff0c9b597d65" (UID: "e6455c7f-5be7-447a-816e-ff0c9b597d65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:04:19.106095 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.106066 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6455c7f-5be7-447a-816e-ff0c9b597d65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6455c7f-5be7-447a-816e-ff0c9b597d65" (UID: "e6455c7f-5be7-447a-816e-ff0c9b597d65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:04:19.106330 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.106315 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6455c7f-5be7-447a-816e-ff0c9b597d65-kube-api-access-mww6d" (OuterVolumeSpecName: "kube-api-access-mww6d") pod "e6455c7f-5be7-447a-816e-ff0c9b597d65" (UID: "e6455c7f-5be7-447a-816e-ff0c9b597d65"). InnerVolumeSpecName "kube-api-access-mww6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:04:19.204961 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.204928 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mww6d\" (UniqueName: \"kubernetes.io/projected/e6455c7f-5be7-447a-816e-ff0c9b597d65-kube-api-access-mww6d\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:04:19.204961 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.204957 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6455c7f-5be7-447a-816e-ff0c9b597d65-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:04:19.204961 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.204967 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6455c7f-5be7-447a-816e-ff0c9b597d65-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:04:19.771442 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.771408 2573 generic.go:356] "Generic (PLEG): container finished" podID="e6455c7f-5be7-447a-816e-ff0c9b597d65" containerID="53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15" exitCode=0 Jul 16 16:04:19.771664 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.771468 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" Jul 16 16:04:19.771664 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.771498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" event={"ID":"e6455c7f-5be7-447a-816e-ff0c9b597d65","Type":"ContainerDied","Data":"53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15"} Jul 16 16:04:19.771664 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.771595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m" event={"ID":"e6455c7f-5be7-447a-816e-ff0c9b597d65","Type":"ContainerDied","Data":"2cfeeca6a94caefb684846fdff97750a6f25a8d1903a11cefd0d26787a466256"} Jul 16 16:04:19.771664 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.771649 2573 scope.go:117] "RemoveContainer" containerID="53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15" Jul 16 16:04:19.781364 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.781347 2573 scope.go:117] "RemoveContainer" containerID="53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15" Jul 16 16:04:19.781672 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:04:19.781602 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15\": container with ID starting with 53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15 not found: ID does not exist" containerID="53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15" Jul 16 16:04:19.781672 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.781661 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15"} err="failed to get container status \"53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15\": rpc error: code = NotFound desc = could not find container \"53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15\": container with ID starting with 53508b0b99cd5b40e6413dbc8ad066ccc313dec573f2fcd529edcfb5cfd30a15 not found: ID does not exist" Jul 16 16:04:19.796295 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.796274 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m"] Jul 16 16:04:19.802767 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:19.802748 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef9fd84b-6f54d9dcd9-zdh6m"] Jul 16 16:04:20.384109 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:04:20.384079 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6455c7f-5be7-447a-816e-ff0c9b597d65" path="/var/lib/kubelet/pods/e6455c7f-5be7-447a-816e-ff0c9b597d65/volumes" Jul 16 16:05:46.140558 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:05:46.140527 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 16:05:46.145472 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:05:46.145449 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 16:06:11.709273 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.709241 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn"] Jul 16 16:06:11.709697 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.709683 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6455c7f-5be7-447a-816e-ff0c9b597d65" containerName="maas-api" Jul 16 16:06:11.709748 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.709701 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6455c7f-5be7-447a-816e-ff0c9b597d65" containerName="maas-api" Jul 16 16:06:11.709785 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.709768 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6455c7f-5be7-447a-816e-ff0c9b597d65" containerName="maas-api" Jul 16 16:06:11.712903 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.712882 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:11.715747 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.715729 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b8455d-serving-cert\"" Jul 16 16:06:11.723806 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.723784 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn"] Jul 16 16:06:11.736149 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.736126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:11.736286 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.736170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wmg\" (UniqueName: \"kubernetes.io/projected/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-kube-api-access-b9wmg\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:11.736286 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.736194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:11.837499 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.837461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:11.837686 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.837517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wmg\" (UniqueName: \"kubernetes.io/projected/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-kube-api-access-b9wmg\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:11.837686 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.837549 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:11.837686 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:06:11.837653 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-serving-cert: secret "maas-api-e2e-inf-a-b8455d-serving-cert" not found Jul 16 16:06:11.837799 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:06:11.837733 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls podName:a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70 nodeName:}" failed. No retries permitted until 2026-07-16 16:06:12.33771247 +0000 UTC m=+1826.776239283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls") pod "maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" (UID: "a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70") : secret "maas-api-e2e-inf-a-b8455d-serving-cert" not found Jul 16 16:06:11.838220 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.838202 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:11.847504 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:11.847483 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wmg\" (UniqueName: \"kubernetes.io/projected/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-kube-api-access-b9wmg\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:12.342570 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:12.342537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:12.344950 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:12.344923 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls\") pod \"maas-api-e2e-inf-a-b8455d-b674565f7-6grjn\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:12.624354 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:12.624271 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:12.767380 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:12.767342 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn"] Jul 16 16:06:12.769753 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:06:12.769724 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e06b11_2ece_4c3c_9a4c_bbf4fa5d8a70.slice/crio-46ae4e79abf3a9748393593feb26c64bb315beab5c051b95f9ab1ec28db0f3d2 WatchSource:0}: Error finding container 46ae4e79abf3a9748393593feb26c64bb315beab5c051b95f9ab1ec28db0f3d2: Status 404 returned error can't find the container with id 46ae4e79abf3a9748393593feb26c64bb315beab5c051b95f9ab1ec28db0f3d2 Jul 16 16:06:12.771218 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:12.771195 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:06:13.332451 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:13.332418 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" event={"ID":"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70","Type":"ContainerStarted","Data":"46ae4e79abf3a9748393593feb26c64bb315beab5c051b95f9ab1ec28db0f3d2"} Jul 16 16:06:16.349397 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:16.349358 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" event={"ID":"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70","Type":"ContainerStarted","Data":"26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e"} Jul 16 16:06:16.349870 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:16.349444 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:16.369760 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:16.369711 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" podStartSLOduration=2.3855990240000002 podStartE2EDuration="5.369696684s" totalImagesPullingTime="2.98409766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:06:16.368860826 +0000 UTC m=+1830.807387654" watchObservedRunningTime="2026-07-16 16:06:16.369696684 +0000 UTC m=+1830.808223510" Jul 16 16:06:19.733971 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.733931 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb"] Jul 16 16:06:19.738047 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.738024 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.740990 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.740970 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7b7961-serving-cert\"" Jul 16 16:06:19.754954 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.754921 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb"] Jul 16 16:06:19.812602 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.812570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wtr\" (UniqueName: \"kubernetes.io/projected/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-kube-api-access-h4wtr\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.812768 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.812711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.812768 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.812750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.913894 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.913858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.914047 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.913903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.914047 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.913946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wtr\" (UniqueName: \"kubernetes.io/projected/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-kube-api-access-h4wtr\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.914660 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.914608 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.916278 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.916250 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:19.923535 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:19.923510 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wtr\" (UniqueName: \"kubernetes.io/projected/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-kube-api-access-h4wtr\") pod \"maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:20.048839 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:20.048804 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:20.198141 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:20.198048 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb"] Jul 16 16:06:20.200368 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:06:20.200337 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd31809_f1fb_4a08_a5e2_a34a8f415ac6.slice/crio-67188292ce78e1d5e1ad11a49978822f87a7b6dca15ba0913a9895e421c7e6e7 WatchSource:0}: Error finding container 67188292ce78e1d5e1ad11a49978822f87a7b6dca15ba0913a9895e421c7e6e7: Status 404 returned error can't find the container with id 67188292ce78e1d5e1ad11a49978822f87a7b6dca15ba0913a9895e421c7e6e7 Jul 16 16:06:20.371277 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:20.371191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" event={"ID":"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6","Type":"ContainerStarted","Data":"67188292ce78e1d5e1ad11a49978822f87a7b6dca15ba0913a9895e421c7e6e7"} Jul 16 16:06:22.361173 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:22.361142 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:06:23.387544 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:23.387515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" event={"ID":"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6","Type":"ContainerStarted","Data":"f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116"} Jul 16 16:06:23.387958 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:23.387625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:23.410395 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:23.410356 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" podStartSLOduration=2.082622179 podStartE2EDuration="4.410342104s" totalImagesPullingTime="2.327719925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:06:23.408795405 +0000 UTC m=+1837.847322229" watchObservedRunningTime="2026-07-16 16:06:23.410342104 +0000 UTC m=+1837.848868927" Jul 16 16:06:29.397262 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:29.397227 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:06:46.222081 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:06:46.222046 2573 scope.go:117] "RemoveContainer" containerID="37692f9987ee92d1cf2da21e358a0465ccd89a5eed3fafded119cc886258e586" Jul 16 16:07:40.281890 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.281693 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn"] Jul 16 16:07:40.282650 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.282600 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" podUID="a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" containerName="maas-api" containerID="cri-o://26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e" gracePeriod=30 Jul 16 16:07:40.532477 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.532418 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:07:40.663102 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.663063 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls\") pod \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " Jul 16 16:07:40.663269 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.663130 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9wmg\" (UniqueName: \"kubernetes.io/projected/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-kube-api-access-b9wmg\") pod \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " Jul 16 16:07:40.663269 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.663207 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-openshift-service-ca\") pod \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\" (UID: \"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70\") " Jul 16 16:07:40.663560 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.663534 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" (UID: "a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:07:40.665164 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.665144 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-kube-api-access-b9wmg" (OuterVolumeSpecName: "kube-api-access-b9wmg") pod "a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" (UID: "a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70"). InnerVolumeSpecName "kube-api-access-b9wmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:07:40.665241 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.665222 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" (UID: "a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:07:40.764797 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.764763 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:07:40.764797 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.764793 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9wmg\" (UniqueName: \"kubernetes.io/projected/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-kube-api-access-b9wmg\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:07:40.764797 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.764803 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:07:40.785002 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.784924 2573 generic.go:356] "Generic (PLEG): container finished" podID="a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" containerID="26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e" exitCode=0 Jul 16 16:07:40.785002 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.784978 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" Jul 16 16:07:40.785002 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.784993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" event={"ID":"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70","Type":"ContainerDied","Data":"26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e"} Jul 16 16:07:40.785199 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.785044 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn" event={"ID":"a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70","Type":"ContainerDied","Data":"46ae4e79abf3a9748393593feb26c64bb315beab5c051b95f9ab1ec28db0f3d2"} Jul 16 16:07:40.785199 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.785075 2573 scope.go:117] "RemoveContainer" containerID="26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e" Jul 16 16:07:40.794274 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.794257 2573 scope.go:117] "RemoveContainer" containerID="26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e" Jul 16 16:07:40.794512 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:07:40.794490 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e\": container with ID starting with 26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e not found: ID does not exist" containerID="26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e" Jul 16 16:07:40.794560 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.794520 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e"} err="failed to get container status \"26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e\": rpc error: code = NotFound desc = could not find container \"26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e\": container with ID starting with 26a86d4d695cda95f2cc5bb808913d9b04a147313cd538f0ec722b656831688e not found: ID does not exist" Jul 16 16:07:40.825030 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.825003 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn"] Jul 16 16:07:40.838252 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:40.838230 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b8455d-b674565f7-6grjn"] Jul 16 16:07:42.383510 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:07:42.383475 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" path="/var/lib/kubelet/pods/a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70/volumes" Jul 16 16:08:23.743941 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:23.743903 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb"] Jul 16 16:08:23.744566 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:23.744141 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" podUID="6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" containerName="maas-api" containerID="cri-o://f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116" gracePeriod=30 Jul 16 16:08:24.191203 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.191179 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:08:24.264916 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.264888 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-openshift-service-ca\") pod \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " Jul 16 16:08:24.265050 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.264948 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-maas-api-tls\") pod \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " Jul 16 16:08:24.265092 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.265058 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4wtr\" (UniqueName: \"kubernetes.io/projected/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-kube-api-access-h4wtr\") pod \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\" (UID: \"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6\") " Jul 16 16:08:24.265274 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.265251 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" (UID: "6bd31809-f1fb-4a08-a5e2-a34a8f415ac6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:08:24.265454 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.265438 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:08:24.267011 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.266980 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-kube-api-access-h4wtr" (OuterVolumeSpecName: "kube-api-access-h4wtr") pod "6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" (UID: "6bd31809-f1fb-4a08-a5e2-a34a8f415ac6"). InnerVolumeSpecName "kube-api-access-h4wtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:08:24.267107 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.267072 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" (UID: "6bd31809-f1fb-4a08-a5e2-a34a8f415ac6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:08:24.366342 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.366314 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:08:24.366342 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:24.366342 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4wtr\" (UniqueName: \"kubernetes.io/projected/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6-kube-api-access-h4wtr\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:08:25.002684 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.002648 2573 generic.go:356] "Generic (PLEG): container finished" podID="6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" containerID="f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116" exitCode=0 Jul 16 16:08:25.003174 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.002701 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" Jul 16 16:08:25.003174 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.002724 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" event={"ID":"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6","Type":"ContainerDied","Data":"f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116"} Jul 16 16:08:25.003174 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.002768 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb" event={"ID":"6bd31809-f1fb-4a08-a5e2-a34a8f415ac6","Type":"ContainerDied","Data":"67188292ce78e1d5e1ad11a49978822f87a7b6dca15ba0913a9895e421c7e6e7"} Jul 16 16:08:25.003174 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.002789 2573 scope.go:117] "RemoveContainer" containerID="f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116" Jul 16 16:08:25.011539 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.011521 2573 scope.go:117] "RemoveContainer" containerID="f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116" Jul 16 16:08:25.011811 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:08:25.011791 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116\": container with ID starting with f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116 not found: ID does not exist" containerID="f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116" Jul 16 16:08:25.011888 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.011821 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116"} err="failed to get container status \"f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116\": rpc error: code = NotFound desc = could not find container \"f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116\": container with ID starting with f6bc4c33112f855459518216ce69a5d819a63cb0c0a1fa65d1d55b86535e0116 not found: ID does not exist" Jul 16 16:08:25.022121 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.022101 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb"] Jul 16 16:08:25.026879 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:25.026857 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7961-64b5959f88-pr7sb"] Jul 16 16:08:26.383713 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:26.383679 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" path="/var/lib/kubelet/pods/6bd31809-f1fb-4a08-a5e2-a34a8f415ac6/volumes" Jul 16 16:08:34.639376 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.639343 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf"] Jul 16 16:08:34.639985 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.639965 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" containerName="maas-api" Jul 16 16:08:34.640030 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.639989 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" containerName="maas-api" Jul 16 16:08:34.640030 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.640007 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" containerName="maas-api" Jul 16 16:08:34.640030 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.640015 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" containerName="maas-api" Jul 16 16:08:34.640146 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.640134 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e06b11-2ece-4c3c-9a4c-bbf4fa5d8a70" containerName="maas-api" Jul 16 16:08:34.640186 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.640153 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd31809-f1fb-4a08-a5e2-a34a8f415ac6" containerName="maas-api" Jul 16 16:08:34.645774 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.645743 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:34.649255 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.649225 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bb0c28-serving-cert\"" Jul 16 16:08:34.660747 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.660719 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf"] Jul 16 16:08:34.765081 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.765046 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:34.765247 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.765140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfdc0b6-5b69-462e-a63f-c2c34354868c-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:34.765247 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.765172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvtm\" (UniqueName: \"kubernetes.io/projected/6cfdc0b6-5b69-462e-a63f-c2c34354868c-kube-api-access-fqvtm\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:34.866581 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.866544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfdc0b6-5b69-462e-a63f-c2c34354868c-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:34.866782 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.866592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvtm\" (UniqueName: \"kubernetes.io/projected/6cfdc0b6-5b69-462e-a63f-c2c34354868c-kube-api-access-fqvtm\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:34.866782 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.866722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:34.866889 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:08:34.866847 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-serving-cert: secret "maas-api-e2e-api-a-bb0c28-serving-cert" not found Jul 16 16:08:34.866945 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:08:34.866932 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls podName:6cfdc0b6-5b69-462e-a63f-c2c34354868c nodeName:}" failed. No retries permitted until 2026-07-16 16:08:35.366910895 +0000 UTC m=+1969.805437703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls") pod "maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" (UID: "6cfdc0b6-5b69-462e-a63f-c2c34354868c") : secret "maas-api-e2e-api-a-bb0c28-serving-cert" not found Jul 16 16:08:34.867193 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.867173 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfdc0b6-5b69-462e-a63f-c2c34354868c-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:34.881419 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:34.881389 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvtm\" (UniqueName: \"kubernetes.io/projected/6cfdc0b6-5b69-462e-a63f-c2c34354868c-kube-api-access-fqvtm\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:35.371119 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:35.371083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:35.373505 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:35.373479 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls\") pod \"maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:35.562925 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:35.562881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:35.702544 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:35.702520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf"] Jul 16 16:08:35.704459 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:08:35.704433 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfdc0b6_5b69_462e_a63f_c2c34354868c.slice/crio-86d1875dec758d0e52636e8fdd3f7852d282472c10d74f4d0827ec02c47363e3 WatchSource:0}: Error finding container 86d1875dec758d0e52636e8fdd3f7852d282472c10d74f4d0827ec02c47363e3: Status 404 returned error can't find the container with id 86d1875dec758d0e52636e8fdd3f7852d282472c10d74f4d0827ec02c47363e3 Jul 16 16:08:36.058722 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:36.058681 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" event={"ID":"6cfdc0b6-5b69-462e-a63f-c2c34354868c","Type":"ContainerStarted","Data":"86d1875dec758d0e52636e8fdd3f7852d282472c10d74f4d0827ec02c47363e3"} Jul 16 16:08:39.074870 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:39.074830 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" event={"ID":"6cfdc0b6-5b69-462e-a63f-c2c34354868c","Type":"ContainerStarted","Data":"f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810"} Jul 16 16:08:39.075335 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:39.074939 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:39.095244 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:39.095199 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" podStartSLOduration=2.102056407 podStartE2EDuration="5.095187178s" totalImagesPullingTime="2.993130771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:08:39.093237508 +0000 UTC m=+1973.531764330" watchObservedRunningTime="2026-07-16 16:08:39.095187178 +0000 UTC m=+1973.533714001" Jul 16 16:08:42.752420 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:42.752387 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq"] Jul 16 16:08:42.756480 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:42.756458 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:42.759797 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:42.759773 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-22d3cd-serving-cert\"" Jul 16 16:08:42.785598 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:42.785572 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq"] Jul 16 16:08:42.946214 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:42.946175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85b25-bb5e-44a4-a358-c6a5b35c3889-openshift-service-ca\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:42.946400 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:42.946239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec85b25-bb5e-44a4-a358-c6a5b35c3889-maas-api-tls\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:42.946400 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:42.946320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgfvz\" (UniqueName: \"kubernetes.io/projected/dec85b25-bb5e-44a4-a358-c6a5b35c3889-kube-api-access-cgfvz\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:43.047118 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:43.047083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85b25-bb5e-44a4-a358-c6a5b35c3889-openshift-service-ca\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:43.047303 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:43.047152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec85b25-bb5e-44a4-a358-c6a5b35c3889-maas-api-tls\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:43.047303 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:43.047181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgfvz\" (UniqueName: \"kubernetes.io/projected/dec85b25-bb5e-44a4-a358-c6a5b35c3889-kube-api-access-cgfvz\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:43.047821 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:43.047800 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85b25-bb5e-44a4-a358-c6a5b35c3889-openshift-service-ca\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:43.049606 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:43.049583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec85b25-bb5e-44a4-a358-c6a5b35c3889-maas-api-tls\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:43.061569 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:43.061541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgfvz\" (UniqueName: \"kubernetes.io/projected/dec85b25-bb5e-44a4-a358-c6a5b35c3889-kube-api-access-cgfvz\") pod \"maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:43.066209 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:43.066187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:43.207802 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:43.207774 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq"] Jul 16 16:08:43.209874 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:08:43.209849 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec85b25_bb5e_44a4_a358_c6a5b35c3889.slice/crio-4f938b76a6c3233570ead6b33520417cb7b2955bd2eb07565b30cb548b7ec72d WatchSource:0}: Error finding container 4f938b76a6c3233570ead6b33520417cb7b2955bd2eb07565b30cb548b7ec72d: Status 404 returned error can't find the container with id 4f938b76a6c3233570ead6b33520417cb7b2955bd2eb07565b30cb548b7ec72d Jul 16 16:08:44.104346 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:44.104299 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" event={"ID":"dec85b25-bb5e-44a4-a358-c6a5b35c3889","Type":"ContainerStarted","Data":"4f938b76a6c3233570ead6b33520417cb7b2955bd2eb07565b30cb548b7ec72d"} Jul 16 16:08:45.086678 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:45.086647 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:46.115164 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:46.115127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" event={"ID":"dec85b25-bb5e-44a4-a358-c6a5b35c3889","Type":"ContainerStarted","Data":"c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb"} Jul 16 16:08:46.115593 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:46.115248 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:46.136130 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:46.136090 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" podStartSLOduration=1.756907936 podStartE2EDuration="4.136076232s" totalImagesPullingTime="2.379168296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:08:46.133593777 +0000 UTC m=+1980.572120600" watchObservedRunningTime="2026-07-16 16:08:46.136076232 +0000 UTC m=+1980.574603052" Jul 16 16:08:52.124732 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:52.124700 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:08:56.391645 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.391597 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf"] Jul 16 16:08:56.392247 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.391886 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" podUID="6cfdc0b6-5b69-462e-a63f-c2c34354868c" containerName="maas-api" containerID="cri-o://f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810" gracePeriod=30 Jul 16 16:08:56.639889 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.639860 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:56.668704 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.668637 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfdc0b6-5b69-462e-a63f-c2c34354868c-openshift-service-ca\") pod \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " Jul 16 16:08:56.668844 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.668709 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqvtm\" (UniqueName: \"kubernetes.io/projected/6cfdc0b6-5b69-462e-a63f-c2c34354868c-kube-api-access-fqvtm\") pod \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " Jul 16 16:08:56.668844 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.668761 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls\") pod \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\" (UID: \"6cfdc0b6-5b69-462e-a63f-c2c34354868c\") " Jul 16 16:08:56.668978 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.668943 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfdc0b6-5b69-462e-a63f-c2c34354868c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6cfdc0b6-5b69-462e-a63f-c2c34354868c" (UID: "6cfdc0b6-5b69-462e-a63f-c2c34354868c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:08:56.670940 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.670903 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfdc0b6-5b69-462e-a63f-c2c34354868c-kube-api-access-fqvtm" (OuterVolumeSpecName: "kube-api-access-fqvtm") pod "6cfdc0b6-5b69-462e-a63f-c2c34354868c" (UID: "6cfdc0b6-5b69-462e-a63f-c2c34354868c"). InnerVolumeSpecName "kube-api-access-fqvtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:08:56.671062 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.670950 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6cfdc0b6-5b69-462e-a63f-c2c34354868c" (UID: "6cfdc0b6-5b69-462e-a63f-c2c34354868c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:08:56.770110 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.770085 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfdc0b6-5b69-462e-a63f-c2c34354868c-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:08:56.770110 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.770110 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqvtm\" (UniqueName: \"kubernetes.io/projected/6cfdc0b6-5b69-462e-a63f-c2c34354868c-kube-api-access-fqvtm\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:08:56.770295 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:56.770119 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfdc0b6-5b69-462e-a63f-c2c34354868c-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:08:57.171684 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.171652 2573 generic.go:356] "Generic (PLEG): container finished" podID="6cfdc0b6-5b69-462e-a63f-c2c34354868c" containerID="f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810" exitCode=0 Jul 16 16:08:57.171859 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.171703 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" Jul 16 16:08:57.171859 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.171738 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" event={"ID":"6cfdc0b6-5b69-462e-a63f-c2c34354868c","Type":"ContainerDied","Data":"f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810"} Jul 16 16:08:57.171859 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.171787 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf" event={"ID":"6cfdc0b6-5b69-462e-a63f-c2c34354868c","Type":"ContainerDied","Data":"86d1875dec758d0e52636e8fdd3f7852d282472c10d74f4d0827ec02c47363e3"} Jul 16 16:08:57.171859 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.171812 2573 scope.go:117] "RemoveContainer" containerID="f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810" Jul 16 16:08:57.180525 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.180510 2573 scope.go:117] "RemoveContainer" containerID="f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810" Jul 16 16:08:57.180805 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:08:57.180787 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810\": container with ID starting with f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810 not found: ID does not exist" containerID="f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810" Jul 16 16:08:57.180860 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.180813 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810"} err="failed to get container status \"f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810\": rpc error: code = NotFound desc = could not find container \"f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810\": container with ID starting with f20c3e4333d8d5b73d20d6072c1e79bc13a501e4aab47934d5fe89343c398810 not found: ID does not exist" Jul 16 16:08:57.206114 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.206090 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf"] Jul 16 16:08:57.209404 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:57.209384 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb0c28-cbff8d7f-zhghf"] Jul 16 16:08:58.384673 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:08:58.384606 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cfdc0b6-5b69-462e-a63f-c2c34354868c" path="/var/lib/kubelet/pods/6cfdc0b6-5b69-462e-a63f-c2c34354868c/volumes" Jul 16 16:09:04.447263 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.447232 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq"] Jul 16 16:09:04.448030 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.447998 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" podUID="dec85b25-bb5e-44a4-a358-c6a5b35c3889" containerName="maas-api" containerID="cri-o://c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb" gracePeriod=30 Jul 16 16:09:04.703365 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.703309 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:09:04.846604 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.846578 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85b25-bb5e-44a4-a358-c6a5b35c3889-openshift-service-ca\") pod \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " Jul 16 16:09:04.846790 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.846643 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgfvz\" (UniqueName: \"kubernetes.io/projected/dec85b25-bb5e-44a4-a358-c6a5b35c3889-kube-api-access-cgfvz\") pod \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " Jul 16 16:09:04.846790 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.846741 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec85b25-bb5e-44a4-a358-c6a5b35c3889-maas-api-tls\") pod \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\" (UID: \"dec85b25-bb5e-44a4-a358-c6a5b35c3889\") " Jul 16 16:09:04.846914 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.846894 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec85b25-bb5e-44a4-a358-c6a5b35c3889-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dec85b25-bb5e-44a4-a358-c6a5b35c3889" (UID: "dec85b25-bb5e-44a4-a358-c6a5b35c3889"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:09:04.847081 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.847063 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85b25-bb5e-44a4-a358-c6a5b35c3889-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:09:04.848661 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.848641 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec85b25-bb5e-44a4-a358-c6a5b35c3889-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dec85b25-bb5e-44a4-a358-c6a5b35c3889" (UID: "dec85b25-bb5e-44a4-a358-c6a5b35c3889"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:09:04.848661 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.848644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec85b25-bb5e-44a4-a358-c6a5b35c3889-kube-api-access-cgfvz" (OuterVolumeSpecName: "kube-api-access-cgfvz") pod "dec85b25-bb5e-44a4-a358-c6a5b35c3889" (UID: "dec85b25-bb5e-44a4-a358-c6a5b35c3889"). InnerVolumeSpecName "kube-api-access-cgfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:09:04.947976 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.947938 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgfvz\" (UniqueName: \"kubernetes.io/projected/dec85b25-bb5e-44a4-a358-c6a5b35c3889-kube-api-access-cgfvz\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:09:04.947976 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:04.947968 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec85b25-bb5e-44a4-a358-c6a5b35c3889-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:09:05.211543 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.211503 2573 generic.go:356] "Generic (PLEG): container finished" podID="dec85b25-bb5e-44a4-a358-c6a5b35c3889" containerID="c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb" exitCode=0 Jul 16 16:09:05.211752 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.211557 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" Jul 16 16:09:05.211752 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.211556 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" event={"ID":"dec85b25-bb5e-44a4-a358-c6a5b35c3889","Type":"ContainerDied","Data":"c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb"} Jul 16 16:09:05.211752 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.211640 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq" event={"ID":"dec85b25-bb5e-44a4-a358-c6a5b35c3889","Type":"ContainerDied","Data":"4f938b76a6c3233570ead6b33520417cb7b2955bd2eb07565b30cb548b7ec72d"} Jul 16 16:09:05.218721 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.211898 2573 scope.go:117] "RemoveContainer" containerID="c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb" Jul 16 16:09:05.228152 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.228133 2573 scope.go:117] "RemoveContainer" containerID="c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb" Jul 16 16:09:05.228391 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:09:05.228374 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb\": container with ID starting with c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb not found: ID does not exist" containerID="c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb" Jul 16 16:09:05.228438 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.228399 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb"} err="failed to get container status \"c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb\": rpc error: code = NotFound desc = could not find container \"c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb\": container with ID starting with c0fbcb70cf421de21db32d12525c1666d6057f546b20c906728654edd06be0fb not found: ID does not exist" Jul 16 16:09:05.241934 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.241913 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq"] Jul 16 16:09:05.245876 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:05.245856 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22d3cd-7778df9dd5-5bcgq"] Jul 16 16:09:06.383972 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:06.383943 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dec85b25-bb5e-44a4-a358-c6a5b35c3889" path="/var/lib/kubelet/pods/dec85b25-bb5e-44a4-a358-c6a5b35c3889/volumes" Jul 16 16:09:14.714150 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.714118 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb"] Jul 16 16:09:14.714535 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.714521 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cfdc0b6-5b69-462e-a63f-c2c34354868c" containerName="maas-api" Jul 16 16:09:14.714578 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.714537 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cfdc0b6-5b69-462e-a63f-c2c34354868c" containerName="maas-api" Jul 16 16:09:14.714578 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.714552 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec85b25-bb5e-44a4-a358-c6a5b35c3889" containerName="maas-api" Jul 16 16:09:14.714578 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.714558 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec85b25-bb5e-44a4-a358-c6a5b35c3889" containerName="maas-api" Jul 16 16:09:14.714692 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.714649 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cfdc0b6-5b69-462e-a63f-c2c34354868c" containerName="maas-api" Jul 16 16:09:14.714692 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.714659 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dec85b25-bb5e-44a4-a358-c6a5b35c3889" containerName="maas-api" Jul 16 16:09:14.719180 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.719162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:14.722831 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.722815 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-83e7d4-serving-cert\"" Jul 16 16:09:14.744483 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.744460 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb"] Jul 16 16:09:14.843084 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.843049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:14.843237 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.843163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5wl\" (UniqueName: \"kubernetes.io/projected/29e13e71-c85c-4f84-861a-9154091eade2-kube-api-access-cm5wl\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:14.843237 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.843195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e13e71-c85c-4f84-861a-9154091eade2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:14.944524 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.944487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:14.944751 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.944588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5wl\" (UniqueName: \"kubernetes.io/projected/29e13e71-c85c-4f84-861a-9154091eade2-kube-api-access-cm5wl\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:14.944751 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.944642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e13e71-c85c-4f84-861a-9154091eade2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:14.944751 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:09:14.944683 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-serving-cert: secret "maas-api-e2e-shared-a-83e7d4-serving-cert" not found Jul 16 16:09:14.944914 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:09:14.944785 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls podName:29e13e71-c85c-4f84-861a-9154091eade2 nodeName:}" failed. No retries permitted until 2026-07-16 16:09:15.444762021 +0000 UTC m=+2009.883288840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls") pod "maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" (UID: "29e13e71-c85c-4f84-861a-9154091eade2") : secret "maas-api-e2e-shared-a-83e7d4-serving-cert" not found Jul 16 16:09:14.945204 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.945186 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e13e71-c85c-4f84-861a-9154091eade2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:14.958400 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:14.958368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5wl\" (UniqueName: \"kubernetes.io/projected/29e13e71-c85c-4f84-861a-9154091eade2-kube-api-access-cm5wl\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:15.447878 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:15.447842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:15.450520 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:15.450494 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls\") pod \"maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:15.628950 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:15.628921 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:15.766889 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:15.766858 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb"] Jul 16 16:09:15.770415 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:09:15.770386 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e13e71_c85c_4f84_861a_9154091eade2.slice/crio-610cc552bb950fbe916547dd2ac61e78bd683f5fc98e924d959da000096d2f25 WatchSource:0}: Error finding container 610cc552bb950fbe916547dd2ac61e78bd683f5fc98e924d959da000096d2f25: Status 404 returned error can't find the container with id 610cc552bb950fbe916547dd2ac61e78bd683f5fc98e924d959da000096d2f25 Jul 16 16:09:16.274096 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:16.274060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" event={"ID":"29e13e71-c85c-4f84-861a-9154091eade2","Type":"ContainerStarted","Data":"79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f"} Jul 16 16:09:16.274271 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:16.274122 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" event={"ID":"29e13e71-c85c-4f84-861a-9154091eade2","Type":"ContainerStarted","Data":"610cc552bb950fbe916547dd2ac61e78bd683f5fc98e924d959da000096d2f25"} Jul 16 16:09:16.274271 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:16.274158 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:16.299573 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:16.299529 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" podStartSLOduration=1.960491837 podStartE2EDuration="2.299513223s" totalImagesPullingTime="339.021386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:09:16.299139658 +0000 UTC m=+2010.737666482" watchObservedRunningTime="2026-07-16 16:09:16.299513223 +0000 UTC m=+2010.738040045" Jul 16 16:09:22.283711 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:22.283677 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:09:22.910854 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:22.910818 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz"] Jul 16 16:09:22.914502 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:22.914479 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:22.917087 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:22.917067 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-590f9a-serving-cert\"" Jul 16 16:09:22.925845 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:22.925823 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz"] Jul 16 16:09:23.017785 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.017760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a41b94b-3e05-43b7-b5a5-a182d766d5af-maas-api-tls\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.017946 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.017808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvggl\" (UniqueName: \"kubernetes.io/projected/9a41b94b-3e05-43b7-b5a5-a182d766d5af-kube-api-access-gvggl\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.017946 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.017914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41b94b-3e05-43b7-b5a5-a182d766d5af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.119193 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.119162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41b94b-3e05-43b7-b5a5-a182d766d5af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.119341 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.119261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a41b94b-3e05-43b7-b5a5-a182d766d5af-maas-api-tls\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.119341 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.119308 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvggl\" (UniqueName: \"kubernetes.io/projected/9a41b94b-3e05-43b7-b5a5-a182d766d5af-kube-api-access-gvggl\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.119908 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.119884 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41b94b-3e05-43b7-b5a5-a182d766d5af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.121651 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.121628 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a41b94b-3e05-43b7-b5a5-a182d766d5af-maas-api-tls\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.127982 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.127961 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvggl\" (UniqueName: \"kubernetes.io/projected/9a41b94b-3e05-43b7-b5a5-a182d766d5af-kube-api-access-gvggl\") pod \"maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.225707 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.225640 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:23.361964 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:23.361934 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz"] Jul 16 16:09:23.364138 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:09:23.364111 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a41b94b_3e05_43b7_b5a5_a182d766d5af.slice/crio-5ff7413c5d5a40d0f8f226e534d150a5652dadc57dffe90697f1a990bf2002b3 WatchSource:0}: Error finding container 5ff7413c5d5a40d0f8f226e534d150a5652dadc57dffe90697f1a990bf2002b3: Status 404 returned error can't find the container with id 5ff7413c5d5a40d0f8f226e534d150a5652dadc57dffe90697f1a990bf2002b3 Jul 16 16:09:24.313794 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:24.313757 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" event={"ID":"9a41b94b-3e05-43b7-b5a5-a182d766d5af","Type":"ContainerStarted","Data":"8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64"} Jul 16 16:09:24.313984 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:24.313813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" event={"ID":"9a41b94b-3e05-43b7-b5a5-a182d766d5af","Type":"ContainerStarted","Data":"5ff7413c5d5a40d0f8f226e534d150a5652dadc57dffe90697f1a990bf2002b3"} Jul 16 16:09:24.313984 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:24.313833 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:30.324197 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:30.324170 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:09:30.347792 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:09:30.347746 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" podStartSLOduration=8.033720158 podStartE2EDuration="8.347731349s" totalImagesPullingTime="314.011191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:09:24.33613949 +0000 UTC m=+2018.774666314" watchObservedRunningTime="2026-07-16 16:09:30.347731349 +0000 UTC m=+2024.786258172" Jul 16 16:10:46.178557 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:10:46.178481 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 16:10:46.184446 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:10:46.184424 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 16:15:00.136133 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.136102 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc"] Jul 16 16:15:00.140093 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.140073 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj"] Jul 16 16:15:00.140214 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.140184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" Jul 16 16:15:00.143191 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.143171 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8"] Jul 16 16:15:00.143307 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.143279 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" Jul 16 16:15:00.146535 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.146514 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" Jul 16 16:15:00.148368 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.148347 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc"] Jul 16 16:15:00.151133 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.151112 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj"] Jul 16 16:15:00.153674 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.153654 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8"] Jul 16 16:15:00.184306 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.184282 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6jm\" (UniqueName: \"kubernetes.io/projected/05635f40-5b57-46ac-b73c-a76dc201130f-kube-api-access-hs6jm\") pod \"maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc\" (UID: \"05635f40-5b57-46ac-b73c-a76dc201130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" Jul 16 16:15:00.285044 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.285016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54tdj\" (UniqueName: \"kubernetes.io/projected/3b609a42-4716-4b02-99ed-f5fd74eb659c-kube-api-access-54tdj\") pod \"maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj\" (UID: \"3b609a42-4716-4b02-99ed-f5fd74eb659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" Jul 16 16:15:00.285187 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.285048 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l76q\" (UniqueName: \"kubernetes.io/projected/004c035b-5d55-443e-8b23-c0967139e790-kube-api-access-2l76q\") pod \"maas-api-key-cleanup-29736975-8ccx8\" (UID: \"004c035b-5d55-443e-8b23-c0967139e790\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" Jul 16 16:15:00.285187 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.285082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6jm\" (UniqueName: \"kubernetes.io/projected/05635f40-5b57-46ac-b73c-a76dc201130f-kube-api-access-hs6jm\") pod \"maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc\" (UID: \"05635f40-5b57-46ac-b73c-a76dc201130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" Jul 16 16:15:00.294623 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.294592 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6jm\" (UniqueName: \"kubernetes.io/projected/05635f40-5b57-46ac-b73c-a76dc201130f-kube-api-access-hs6jm\") pod \"maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc\" (UID: \"05635f40-5b57-46ac-b73c-a76dc201130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" Jul 16 16:15:00.386184 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.386115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54tdj\" (UniqueName: \"kubernetes.io/projected/3b609a42-4716-4b02-99ed-f5fd74eb659c-kube-api-access-54tdj\") pod \"maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj\" (UID: \"3b609a42-4716-4b02-99ed-f5fd74eb659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" Jul 16 16:15:00.386184 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.386150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l76q\" (UniqueName: \"kubernetes.io/projected/004c035b-5d55-443e-8b23-c0967139e790-kube-api-access-2l76q\") pod \"maas-api-key-cleanup-29736975-8ccx8\" (UID: \"004c035b-5d55-443e-8b23-c0967139e790\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" Jul 16 16:15:00.396458 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.396427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l76q\" (UniqueName: \"kubernetes.io/projected/004c035b-5d55-443e-8b23-c0967139e790-kube-api-access-2l76q\") pod \"maas-api-key-cleanup-29736975-8ccx8\" (UID: \"004c035b-5d55-443e-8b23-c0967139e790\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" Jul 16 16:15:00.397186 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.397164 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54tdj\" (UniqueName: \"kubernetes.io/projected/3b609a42-4716-4b02-99ed-f5fd74eb659c-kube-api-access-54tdj\") pod \"maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj\" (UID: \"3b609a42-4716-4b02-99ed-f5fd74eb659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" Jul 16 16:15:00.453342 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.453313 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" Jul 16 16:15:00.461628 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.461585 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" Jul 16 16:15:00.468669 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.468647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" Jul 16 16:15:00.617561 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.617524 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc"] Jul 16 16:15:00.620795 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:00.620763 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05635f40_5b57_46ac_b73c_a76dc201130f.slice/crio-7dc1a55902dad40c3fd8aabd7dae1e281ea0c1e19952bcc6863de4820c897f86 WatchSource:0}: Error finding container 7dc1a55902dad40c3fd8aabd7dae1e281ea0c1e19952bcc6863de4820c897f86: Status 404 returned error can't find the container with id 7dc1a55902dad40c3fd8aabd7dae1e281ea0c1e19952bcc6863de4820c897f86 Jul 16 16:15:00.664899 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.664873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8"] Jul 16 16:15:00.666453 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:00.666428 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004c035b_5d55_443e_8b23_c0967139e790.slice/crio-d99605bdf452147c890a6ac95a390daa85832190ad1b3829d6dbed395fe8698d WatchSource:0}: Error finding container d99605bdf452147c890a6ac95a390daa85832190ad1b3829d6dbed395fe8698d: Status 404 returned error can't find the container with id d99605bdf452147c890a6ac95a390daa85832190ad1b3829d6dbed395fe8698d Jul 16 16:15:00.853998 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:00.853970 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj"] Jul 16 16:15:00.856683 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:00.856570 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b609a42_4716_4b02_99ed_f5fd74eb659c.slice/crio-12517eaedbfc7612efce9f4c2373ff55e08d4796d4ed1ab38644c3ec38211006 WatchSource:0}: Error finding container 12517eaedbfc7612efce9f4c2373ff55e08d4796d4ed1ab38644c3ec38211006: Status 404 returned error can't find the container with id 12517eaedbfc7612efce9f4c2373ff55e08d4796d4ed1ab38644c3ec38211006 Jul 16 16:15:01.011101 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:01.011064 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" event={"ID":"3b609a42-4716-4b02-99ed-f5fd74eb659c","Type":"ContainerStarted","Data":"12517eaedbfc7612efce9f4c2373ff55e08d4796d4ed1ab38644c3ec38211006"} Jul 16 16:15:01.012541 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:01.012509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" event={"ID":"05635f40-5b57-46ac-b73c-a76dc201130f","Type":"ContainerStarted","Data":"132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525"} Jul 16 16:15:01.012689 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:01.012558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" event={"ID":"05635f40-5b57-46ac-b73c-a76dc201130f","Type":"ContainerStarted","Data":"7dc1a55902dad40c3fd8aabd7dae1e281ea0c1e19952bcc6863de4820c897f86"} Jul 16 16:15:01.013756 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:01.013721 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" event={"ID":"004c035b-5d55-443e-8b23-c0967139e790","Type":"ContainerStarted","Data":"d99605bdf452147c890a6ac95a390daa85832190ad1b3829d6dbed395fe8698d"} Jul 16 16:15:01.029908 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:01.029866 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" podStartSLOduration=1.029849309 podStartE2EDuration="1.029849309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:15:01.028656712 +0000 UTC m=+2355.467183535" watchObservedRunningTime="2026-07-16 16:15:01.029849309 +0000 UTC m=+2355.468376132" Jul 16 16:15:02.019977 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:02.019845 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" event={"ID":"004c035b-5d55-443e-8b23-c0967139e790","Type":"ContainerStarted","Data":"df6f3d55a457828f5a7ac66f4766d2063254aec44183dae9c83b60653894ee63"} Jul 16 16:15:02.034686 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:02.021336 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" event={"ID":"3b609a42-4716-4b02-99ed-f5fd74eb659c","Type":"ContainerStarted","Data":"81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76"} Jul 16 16:15:02.057538 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:02.057491 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" podStartSLOduration=2.057474187 podStartE2EDuration="2.057474187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:15:02.037956671 +0000 UTC m=+2356.476483494" watchObservedRunningTime="2026-07-16 16:15:02.057474187 +0000 UTC m=+2356.496001011" Jul 16 16:15:02.059808 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:02.059767 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" podStartSLOduration=2.059755836 podStartE2EDuration="2.059755836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:15:02.057034663 +0000 UTC m=+2356.495561489" watchObservedRunningTime="2026-07-16 16:15:02.059755836 +0000 UTC m=+2356.498282659" Jul 16 16:15:03.903139 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:03.903104 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7447-140072614862000"] Jul 16 16:15:03.906728 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:03.906712 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072614862000" Jul 16 16:15:03.910765 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:03.910745 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-jxk2v\"" Jul 16 16:15:03.917236 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:03.917213 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072614862000"] Jul 16 16:15:04.020498 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:04.020463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsm2h\" (UniqueName: \"kubernetes.io/projected/e2f2b89b-69d9-4505-b8e4-2a1f7d426b59-kube-api-access-qsm2h\") pod \"test-curl-7447-140072614862000\" (UID: \"e2f2b89b-69d9-4505-b8e4-2a1f7d426b59\") " pod="openshift-ingress/test-curl-7447-140072614862000" Jul 16 16:15:04.121676 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:04.121642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsm2h\" (UniqueName: \"kubernetes.io/projected/e2f2b89b-69d9-4505-b8e4-2a1f7d426b59-kube-api-access-qsm2h\") pod \"test-curl-7447-140072614862000\" (UID: \"e2f2b89b-69d9-4505-b8e4-2a1f7d426b59\") " pod="openshift-ingress/test-curl-7447-140072614862000" Jul 16 16:15:04.131480 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:04.131451 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsm2h\" (UniqueName: \"kubernetes.io/projected/e2f2b89b-69d9-4505-b8e4-2a1f7d426b59-kube-api-access-qsm2h\") pod \"test-curl-7447-140072614862000\" (UID: \"e2f2b89b-69d9-4505-b8e4-2a1f7d426b59\") " pod="openshift-ingress/test-curl-7447-140072614862000" Jul 16 16:15:04.217231 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:04.217159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072614862000" Jul 16 16:15:04.372913 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:04.372846 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072614862000"] Jul 16 16:15:04.375662 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:04.375627 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2f2b89b_69d9_4505_b8e4_2a1f7d426b59.slice/crio-911ce990b4cafd1a24f2a6c017a1429af878197602f9c87cad033fd01103a24e WatchSource:0}: Error finding container 911ce990b4cafd1a24f2a6c017a1429af878197602f9c87cad033fd01103a24e: Status 404 returned error can't find the container with id 911ce990b4cafd1a24f2a6c017a1429af878197602f9c87cad033fd01103a24e Jul 16 16:15:04.377163 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:04.377142 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:15:05.039789 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:05.039746 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072614862000" event={"ID":"e2f2b89b-69d9-4505-b8e4-2a1f7d426b59","Type":"ContainerStarted","Data":"911ce990b4cafd1a24f2a6c017a1429af878197602f9c87cad033fd01103a24e"} Jul 16 16:15:06.046697 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:06.046661 2573 generic.go:356] "Generic (PLEG): container finished" podID="e2f2b89b-69d9-4505-b8e4-2a1f7d426b59" containerID="051a41e2040f41040d37d23f460405764d888e60c3571ce40121c7e1f1217212" exitCode=0 Jul 16 16:15:06.047090 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:06.046750 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072614862000" event={"ID":"e2f2b89b-69d9-4505-b8e4-2a1f7d426b59","Type":"ContainerDied","Data":"051a41e2040f41040d37d23f460405764d888e60c3571ce40121c7e1f1217212"} Jul 16 16:15:07.203540 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.203512 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072614862000" Jul 16 16:15:07.249117 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.249088 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7447-140072614862000_e2f2b89b-69d9-4505-b8e4-2a1f7d426b59/test-curl-7447-140072614862000/0.log" Jul 16 16:15:07.274605 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.274578 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7447-140072614862000"] Jul 16 16:15:07.279068 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.279040 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7447-140072614862000"] Jul 16 16:15:07.351264 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.351184 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsm2h\" (UniqueName: \"kubernetes.io/projected/e2f2b89b-69d9-4505-b8e4-2a1f7d426b59-kube-api-access-qsm2h\") pod \"e2f2b89b-69d9-4505-b8e4-2a1f7d426b59\" (UID: \"e2f2b89b-69d9-4505-b8e4-2a1f7d426b59\") " Jul 16 16:15:07.353573 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.353543 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f2b89b-69d9-4505-b8e4-2a1f7d426b59-kube-api-access-qsm2h" (OuterVolumeSpecName: "kube-api-access-qsm2h") pod "e2f2b89b-69d9-4505-b8e4-2a1f7d426b59" (UID: "e2f2b89b-69d9-4505-b8e4-2a1f7d426b59"). InnerVolumeSpecName "kube-api-access-qsm2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:07.423816 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.423793 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7447-140072614862256"] Jul 16 16:15:07.424198 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.424185 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f2b89b-69d9-4505-b8e4-2a1f7d426b59" containerName="test-curl-7447-140072614862000" Jul 16 16:15:07.424242 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.424200 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f2b89b-69d9-4505-b8e4-2a1f7d426b59" containerName="test-curl-7447-140072614862000" Jul 16 16:15:07.424306 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.424298 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f2b89b-69d9-4505-b8e4-2a1f7d426b59" containerName="test-curl-7447-140072614862000" Jul 16 16:15:07.427197 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.427178 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072614862256" Jul 16 16:15:07.436741 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.436719 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072614862256"] Jul 16 16:15:07.452894 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.452870 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsm2h\" (UniqueName: \"kubernetes.io/projected/e2f2b89b-69d9-4505-b8e4-2a1f7d426b59-kube-api-access-qsm2h\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:07.553626 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.553586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ztv2\" (UniqueName: \"kubernetes.io/projected/ebcd2d70-451b-44b6-a83f-4baf53ee22a2-kube-api-access-8ztv2\") pod \"test-curl-7447-140072614862256\" (UID: \"ebcd2d70-451b-44b6-a83f-4baf53ee22a2\") " pod="openshift-ingress/test-curl-7447-140072614862256" Jul 16 16:15:07.654773 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.654683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ztv2\" (UniqueName: \"kubernetes.io/projected/ebcd2d70-451b-44b6-a83f-4baf53ee22a2-kube-api-access-8ztv2\") pod \"test-curl-7447-140072614862256\" (UID: \"ebcd2d70-451b-44b6-a83f-4baf53ee22a2\") " pod="openshift-ingress/test-curl-7447-140072614862256" Jul 16 16:15:07.664428 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.664395 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ztv2\" (UniqueName: \"kubernetes.io/projected/ebcd2d70-451b-44b6-a83f-4baf53ee22a2-kube-api-access-8ztv2\") pod \"test-curl-7447-140072614862256\" (UID: \"ebcd2d70-451b-44b6-a83f-4baf53ee22a2\") " pod="openshift-ingress/test-curl-7447-140072614862256" Jul 16 16:15:07.737110 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.737073 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072614862256" Jul 16 16:15:07.873063 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:07.873038 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072614862256"] Jul 16 16:15:07.875105 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:07.875081 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebcd2d70_451b_44b6_a83f_4baf53ee22a2.slice/crio-c306702d164cacb5475be652d77c9191c669974e554db07c84b76fbd8945f526 WatchSource:0}: Error finding container c306702d164cacb5475be652d77c9191c669974e554db07c84b76fbd8945f526: Status 404 returned error can't find the container with id c306702d164cacb5475be652d77c9191c669974e554db07c84b76fbd8945f526 Jul 16 16:15:08.058413 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:08.058381 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="911ce990b4cafd1a24f2a6c017a1429af878197602f9c87cad033fd01103a24e" Jul 16 16:15:08.058413 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:08.058385 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072614862000" Jul 16 16:15:08.059655 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:08.059629 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072614862256" event={"ID":"ebcd2d70-451b-44b6-a83f-4baf53ee22a2","Type":"ContainerStarted","Data":"c306702d164cacb5475be652d77c9191c669974e554db07c84b76fbd8945f526"} Jul 16 16:15:08.071294 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:15:08.071270 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7447-140072614862000\" is forbidden: User \"system:node:ip-10-0-128-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-48.ec2.internal' and this object" podUID="e2f2b89b-69d9-4505-b8e4-2a1f7d426b59" pod="openshift-ingress/test-curl-7447-140072614862000" Jul 16 16:15:08.383827 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:08.383797 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f2b89b-69d9-4505-b8e4-2a1f7d426b59" path="/var/lib/kubelet/pods/e2f2b89b-69d9-4505-b8e4-2a1f7d426b59/volumes" Jul 16 16:15:09.065769 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:09.065734 2573 generic.go:356] "Generic (PLEG): container finished" podID="ebcd2d70-451b-44b6-a83f-4baf53ee22a2" containerID="b6857b83565411f83244c1b5a52789286266f6c94173fe2b9f0c2f9331afb9f7" exitCode=0 Jul 16 16:15:09.065946 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:09.065824 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072614862256" event={"ID":"ebcd2d70-451b-44b6-a83f-4baf53ee22a2","Type":"ContainerDied","Data":"b6857b83565411f83244c1b5a52789286266f6c94173fe2b9f0c2f9331afb9f7"} Jul 16 16:15:10.218551 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.218527 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072614862256" Jul 16 16:15:10.248279 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.248253 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7447-140072614862256_ebcd2d70-451b-44b6-a83f-4baf53ee22a2/test-curl-7447-140072614862256/0.log" Jul 16 16:15:10.276669 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.276634 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7447-140072614862256"] Jul 16 16:15:10.280895 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.280871 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7447-140072614862256"] Jul 16 16:15:10.378812 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.378727 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ztv2\" (UniqueName: \"kubernetes.io/projected/ebcd2d70-451b-44b6-a83f-4baf53ee22a2-kube-api-access-8ztv2\") pod \"ebcd2d70-451b-44b6-a83f-4baf53ee22a2\" (UID: \"ebcd2d70-451b-44b6-a83f-4baf53ee22a2\") " Jul 16 16:15:10.380861 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.380833 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcd2d70-451b-44b6-a83f-4baf53ee22a2-kube-api-access-8ztv2" (OuterVolumeSpecName: "kube-api-access-8ztv2") pod "ebcd2d70-451b-44b6-a83f-4baf53ee22a2" (UID: "ebcd2d70-451b-44b6-a83f-4baf53ee22a2"). InnerVolumeSpecName "kube-api-access-8ztv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:10.384722 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.384695 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebcd2d70-451b-44b6-a83f-4baf53ee22a2" path="/var/lib/kubelet/pods/ebcd2d70-451b-44b6-a83f-4baf53ee22a2/volumes" Jul 16 16:15:10.455232 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.455204 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:10.455866 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.455850 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcd2d70-451b-44b6-a83f-4baf53ee22a2" containerName="test-curl-7447-140072614862256" Jul 16 16:15:10.455915 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.455870 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcd2d70-451b-44b6-a83f-4baf53ee22a2" containerName="test-curl-7447-140072614862256" Jul 16 16:15:10.455967 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.455956 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcd2d70-451b-44b6-a83f-4baf53ee22a2" containerName="test-curl-7447-140072614862256" Jul 16 16:15:10.459440 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.459421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:10.463636 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.463600 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:10.480041 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.480021 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ztv2\" (UniqueName: \"kubernetes.io/projected/ebcd2d70-451b-44b6-a83f-4baf53ee22a2-kube-api-access-8ztv2\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:10.581406 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.581376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76hrd\" (UniqueName: \"kubernetes.io/projected/40c4a036-039c-4edb-870c-f17bed0cd620-kube-api-access-76hrd\") pod \"test-curl-7447-140072613726368\" (UID: \"40c4a036-039c-4edb-870c-f17bed0cd620\") " pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:10.682022 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.681946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76hrd\" (UniqueName: \"kubernetes.io/projected/40c4a036-039c-4edb-870c-f17bed0cd620-kube-api-access-76hrd\") pod \"test-curl-7447-140072613726368\" (UID: \"40c4a036-039c-4edb-870c-f17bed0cd620\") " pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:10.691353 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.691329 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76hrd\" (UniqueName: \"kubernetes.io/projected/40c4a036-039c-4edb-870c-f17bed0cd620-kube-api-access-76hrd\") pod \"test-curl-7447-140072613726368\" (UID: \"40c4a036-039c-4edb-870c-f17bed0cd620\") " pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:10.769599 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.769572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:10.904861 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:10.904835 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:10.907403 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:10.907377 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40c4a036_039c_4edb_870c_f17bed0cd620.slice/crio-008af032e0c4a0aa66fbd9adae6d24a56d4b49e07b93b0ff242ed46571b73689 WatchSource:0}: Error finding container 008af032e0c4a0aa66fbd9adae6d24a56d4b49e07b93b0ff242ed46571b73689: Status 404 returned error can't find the container with id 008af032e0c4a0aa66fbd9adae6d24a56d4b49e07b93b0ff242ed46571b73689 Jul 16 16:15:11.077477 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:11.077446 2573 scope.go:117] "RemoveContainer" containerID="b6857b83565411f83244c1b5a52789286266f6c94173fe2b9f0c2f9331afb9f7" Jul 16 16:15:11.077677 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:11.077453 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072614862256" Jul 16 16:15:11.078768 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:11.078677 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072613726368" event={"ID":"40c4a036-039c-4edb-870c-f17bed0cd620","Type":"ContainerStarted","Data":"008af032e0c4a0aa66fbd9adae6d24a56d4b49e07b93b0ff242ed46571b73689"} Jul 16 16:15:12.087102 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:12.087066 2573 generic.go:356] "Generic (PLEG): container finished" podID="40c4a036-039c-4edb-870c-f17bed0cd620" containerID="2ad72a7a462e950c989d3b9f140e10ebbd35696488a5a6b5f88c799f8dbeb21c" exitCode=0 Jul 16 16:15:12.087485 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:12.087154 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072613726368" event={"ID":"40c4a036-039c-4edb-870c-f17bed0cd620","Type":"ContainerDied","Data":"2ad72a7a462e950c989d3b9f140e10ebbd35696488a5a6b5f88c799f8dbeb21c"} Jul 16 16:15:13.240247 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.240224 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:13.263768 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.263746 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7447-140072613726368_40c4a036-039c-4edb-870c-f17bed0cd620/test-curl-7447-140072613726368/0.log" Jul 16 16:15:13.290018 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.289992 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:13.293912 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.293890 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:13.408975 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.408913 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76hrd\" (UniqueName: \"kubernetes.io/projected/40c4a036-039c-4edb-870c-f17bed0cd620-kube-api-access-76hrd\") pod \"40c4a036-039c-4edb-870c-f17bed0cd620\" (UID: \"40c4a036-039c-4edb-870c-f17bed0cd620\") " Jul 16 16:15:13.410888 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.410858 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c4a036-039c-4edb-870c-f17bed0cd620-kube-api-access-76hrd" (OuterVolumeSpecName: "kube-api-access-76hrd") pod "40c4a036-039c-4edb-870c-f17bed0cd620" (UID: "40c4a036-039c-4edb-870c-f17bed0cd620"). InnerVolumeSpecName "kube-api-access-76hrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:13.423081 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.423059 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:13.423522 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.423509 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c4a036-039c-4edb-870c-f17bed0cd620" containerName="test-curl-7447-140072613726368" Jul 16 16:15:13.423575 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.423524 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c4a036-039c-4edb-870c-f17bed0cd620" containerName="test-curl-7447-140072613726368" Jul 16 16:15:13.423628 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.423608 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c4a036-039c-4edb-870c-f17bed0cd620" containerName="test-curl-7447-140072613726368" Jul 16 16:15:13.426541 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.426526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:13.433946 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.433926 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:13.510790 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.510742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpws\" (UniqueName: \"kubernetes.io/projected/eba76432-3eaa-441c-93a9-84d16645a21f-kube-api-access-2mpws\") pod \"test-curl-7447-140072613726368\" (UID: \"eba76432-3eaa-441c-93a9-84d16645a21f\") " pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:13.510968 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.510921 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76hrd\" (UniqueName: \"kubernetes.io/projected/40c4a036-039c-4edb-870c-f17bed0cd620-kube-api-access-76hrd\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:13.611768 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.611738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpws\" (UniqueName: \"kubernetes.io/projected/eba76432-3eaa-441c-93a9-84d16645a21f-kube-api-access-2mpws\") pod \"test-curl-7447-140072613726368\" (UID: \"eba76432-3eaa-441c-93a9-84d16645a21f\") " pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:13.619935 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.619917 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpws\" (UniqueName: \"kubernetes.io/projected/eba76432-3eaa-441c-93a9-84d16645a21f-kube-api-access-2mpws\") pod \"test-curl-7447-140072613726368\" (UID: \"eba76432-3eaa-441c-93a9-84d16645a21f\") " pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:13.736154 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.736077 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:13.869455 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:13.869424 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:13.872750 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:13.872724 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeba76432_3eaa_441c_93a9_84d16645a21f.slice/crio-83b47da8a97482203440acced719bec2a6fc513858ce120cb1fc0807405974f6 WatchSource:0}: Error finding container 83b47da8a97482203440acced719bec2a6fc513858ce120cb1fc0807405974f6: Status 404 returned error can't find the container with id 83b47da8a97482203440acced719bec2a6fc513858ce120cb1fc0807405974f6 Jul 16 16:15:14.097792 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:14.097756 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:14.097792 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:14.097783 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="008af032e0c4a0aa66fbd9adae6d24a56d4b49e07b93b0ff242ed46571b73689" Jul 16 16:15:14.098877 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:14.098852 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072613726368" event={"ID":"eba76432-3eaa-441c-93a9-84d16645a21f","Type":"ContainerStarted","Data":"83b47da8a97482203440acced719bec2a6fc513858ce120cb1fc0807405974f6"} Jul 16 16:15:14.116802 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:14.116770 2573 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7447-140072613726368" oldPodUID="40c4a036-039c-4edb-870c-f17bed0cd620" podUID="eba76432-3eaa-441c-93a9-84d16645a21f" Jul 16 16:15:14.140912 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:15:14.140882 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40c4a036_039c_4edb_870c_f17bed0cd620.slice\": RecentStats: unable to find data in memory cache]" Jul 16 16:15:14.383771 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:14.383680 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c4a036-039c-4edb-870c-f17bed0cd620" path="/var/lib/kubelet/pods/40c4a036-039c-4edb-870c-f17bed0cd620/volumes" Jul 16 16:15:15.105697 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:15.105661 2573 generic.go:356] "Generic (PLEG): container finished" podID="eba76432-3eaa-441c-93a9-84d16645a21f" containerID="983a8555b306808309fab883e23e2003ae9681335906719658102ab23cdbe04c" exitCode=0 Jul 16 16:15:15.105883 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:15.105744 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072613726368" event={"ID":"eba76432-3eaa-441c-93a9-84d16645a21f","Type":"ContainerDied","Data":"983a8555b306808309fab883e23e2003ae9681335906719658102ab23cdbe04c"} Jul 16 16:15:16.243417 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.243392 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:16.266633 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.266575 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7447-140072613726368_eba76432-3eaa-441c-93a9-84d16645a21f/test-curl-7447-140072613726368/0.log" Jul 16 16:15:16.294404 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.294378 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:16.299855 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.299832 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7447-140072613726368"] Jul 16 16:15:16.335701 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.335679 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mpws\" (UniqueName: \"kubernetes.io/projected/eba76432-3eaa-441c-93a9-84d16645a21f-kube-api-access-2mpws\") pod \"eba76432-3eaa-441c-93a9-84d16645a21f\" (UID: \"eba76432-3eaa-441c-93a9-84d16645a21f\") " Jul 16 16:15:16.337788 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.337758 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba76432-3eaa-441c-93a9-84d16645a21f-kube-api-access-2mpws" (OuterVolumeSpecName: "kube-api-access-2mpws") pod "eba76432-3eaa-441c-93a9-84d16645a21f" (UID: "eba76432-3eaa-441c-93a9-84d16645a21f"). InnerVolumeSpecName "kube-api-access-2mpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:16.384772 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.384715 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba76432-3eaa-441c-93a9-84d16645a21f" path="/var/lib/kubelet/pods/eba76432-3eaa-441c-93a9-84d16645a21f/volumes" Jul 16 16:15:16.427261 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.427235 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:16.427665 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.427653 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba76432-3eaa-441c-93a9-84d16645a21f" containerName="test-curl-7447-140072613726368" Jul 16 16:15:16.427707 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.427666 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba76432-3eaa-441c-93a9-84d16645a21f" containerName="test-curl-7447-140072613726368" Jul 16 16:15:16.427738 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.427727 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba76432-3eaa-441c-93a9-84d16645a21f" containerName="test-curl-7447-140072613726368" Jul 16 16:15:16.430700 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.430684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:16.437756 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.437167 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mpws\" (UniqueName: \"kubernetes.io/projected/eba76432-3eaa-441c-93a9-84d16645a21f-kube-api-access-2mpws\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:16.438734 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.438714 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:16.537899 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.537864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzxm\" (UniqueName: \"kubernetes.io/projected/f848aa33-2af9-49b7-93d0-77cd389c082e-kube-api-access-rdzxm\") pod \"test-curl-7447-140072613727376\" (UID: \"f848aa33-2af9-49b7-93d0-77cd389c082e\") " pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:16.639526 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.639449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzxm\" (UniqueName: \"kubernetes.io/projected/f848aa33-2af9-49b7-93d0-77cd389c082e-kube-api-access-rdzxm\") pod \"test-curl-7447-140072613727376\" (UID: \"f848aa33-2af9-49b7-93d0-77cd389c082e\") " pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:16.649729 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.649702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzxm\" (UniqueName: \"kubernetes.io/projected/f848aa33-2af9-49b7-93d0-77cd389c082e-kube-api-access-rdzxm\") pod \"test-curl-7447-140072613727376\" (UID: \"f848aa33-2af9-49b7-93d0-77cd389c082e\") " pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:16.740386 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.740356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:16.872153 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:16.872118 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:16.875594 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:16.875558 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf848aa33_2af9_49b7_93d0_77cd389c082e.slice/crio-501e004a2e1de1568c41c6ea2198a546e5ac390f9f7165a5186c6e65ad815d15 WatchSource:0}: Error finding container 501e004a2e1de1568c41c6ea2198a546e5ac390f9f7165a5186c6e65ad815d15: Status 404 returned error can't find the container with id 501e004a2e1de1568c41c6ea2198a546e5ac390f9f7165a5186c6e65ad815d15 Jul 16 16:15:17.120893 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:17.120855 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072613727376" event={"ID":"f848aa33-2af9-49b7-93d0-77cd389c082e","Type":"ContainerStarted","Data":"501e004a2e1de1568c41c6ea2198a546e5ac390f9f7165a5186c6e65ad815d15"} Jul 16 16:15:17.121995 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:17.121975 2573 scope.go:117] "RemoveContainer" containerID="983a8555b306808309fab883e23e2003ae9681335906719658102ab23cdbe04c" Jul 16 16:15:17.121995 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:17.121994 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613726368" Jul 16 16:15:18.129745 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:18.129707 2573 generic.go:356] "Generic (PLEG): container finished" podID="f848aa33-2af9-49b7-93d0-77cd389c082e" containerID="32b46b2624511e9fe759202cd7ab5583354fc960f8b2be282ee09eb8aeb2da6e" exitCode=0 Jul 16 16:15:18.130171 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:18.129799 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072613727376" event={"ID":"f848aa33-2af9-49b7-93d0-77cd389c082e","Type":"ContainerDied","Data":"32b46b2624511e9fe759202cd7ab5583354fc960f8b2be282ee09eb8aeb2da6e"} Jul 16 16:15:19.280006 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.279982 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:19.303448 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.303424 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7447-140072613727376_f848aa33-2af9-49b7-93d0-77cd389c082e/test-curl-7447-140072613727376/0.log" Jul 16 16:15:19.329920 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.329897 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:19.333716 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.333692 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:19.468870 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.468784 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdzxm\" (UniqueName: \"kubernetes.io/projected/f848aa33-2af9-49b7-93d0-77cd389c082e-kube-api-access-rdzxm\") pod \"f848aa33-2af9-49b7-93d0-77cd389c082e\" (UID: \"f848aa33-2af9-49b7-93d0-77cd389c082e\") " Jul 16 16:15:19.471084 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.471051 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f848aa33-2af9-49b7-93d0-77cd389c082e-kube-api-access-rdzxm" (OuterVolumeSpecName: "kube-api-access-rdzxm") pod "f848aa33-2af9-49b7-93d0-77cd389c082e" (UID: "f848aa33-2af9-49b7-93d0-77cd389c082e"). InnerVolumeSpecName "kube-api-access-rdzxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:19.471517 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.471496 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:19.471934 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.471922 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f848aa33-2af9-49b7-93d0-77cd389c082e" containerName="test-curl-7447-140072613727376" Jul 16 16:15:19.471970 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.471936 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f848aa33-2af9-49b7-93d0-77cd389c082e" containerName="test-curl-7447-140072613727376" Jul 16 16:15:19.472014 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.472005 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f848aa33-2af9-49b7-93d0-77cd389c082e" containerName="test-curl-7447-140072613727376" Jul 16 16:15:19.474881 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.474867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:19.483743 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.483715 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:19.569923 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.569890 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzxm\" (UniqueName: \"kubernetes.io/projected/f848aa33-2af9-49b7-93d0-77cd389c082e-kube-api-access-rdzxm\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:19.671345 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.671313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5xph\" (UniqueName: \"kubernetes.io/projected/911f56bc-7ce7-41b9-9c77-40f60f174134-kube-api-access-d5xph\") pod \"test-curl-7447-140072613727376\" (UID: \"911f56bc-7ce7-41b9-9c77-40f60f174134\") " pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:19.772664 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.772631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5xph\" (UniqueName: \"kubernetes.io/projected/911f56bc-7ce7-41b9-9c77-40f60f174134-kube-api-access-d5xph\") pod \"test-curl-7447-140072613727376\" (UID: \"911f56bc-7ce7-41b9-9c77-40f60f174134\") " pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:19.782170 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.782135 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5xph\" (UniqueName: \"kubernetes.io/projected/911f56bc-7ce7-41b9-9c77-40f60f174134-kube-api-access-d5xph\") pod \"test-curl-7447-140072613727376\" (UID: \"911f56bc-7ce7-41b9-9c77-40f60f174134\") " pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:19.788039 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.788020 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:19.924055 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:19.924032 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:19.924879 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:19.924857 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod911f56bc_7ce7_41b9_9c77_40f60f174134.slice/crio-f0d22e4725b441ab04315279792de4ca3e087fff70c1d3394cf733b0ca075f51 WatchSource:0}: Error finding container f0d22e4725b441ab04315279792de4ca3e087fff70c1d3394cf733b0ca075f51: Status 404 returned error can't find the container with id f0d22e4725b441ab04315279792de4ca3e087fff70c1d3394cf733b0ca075f51 Jul 16 16:15:20.141997 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:20.141907 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072613727376" event={"ID":"911f56bc-7ce7-41b9-9c77-40f60f174134","Type":"ContainerStarted","Data":"f0d22e4725b441ab04315279792de4ca3e087fff70c1d3394cf733b0ca075f51"} Jul 16 16:15:20.143148 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:20.143122 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:20.143148 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:20.143137 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="501e004a2e1de1568c41c6ea2198a546e5ac390f9f7165a5186c6e65ad815d15" Jul 16 16:15:20.158350 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:20.158322 2573 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7447-140072613727376" oldPodUID="f848aa33-2af9-49b7-93d0-77cd389c082e" podUID="911f56bc-7ce7-41b9-9c77-40f60f174134" Jul 16 16:15:20.384433 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:20.384403 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f848aa33-2af9-49b7-93d0-77cd389c082e" path="/var/lib/kubelet/pods/f848aa33-2af9-49b7-93d0-77cd389c082e/volumes" Jul 16 16:15:21.149950 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:21.149918 2573 generic.go:356] "Generic (PLEG): container finished" podID="911f56bc-7ce7-41b9-9c77-40f60f174134" containerID="d371ad1f56f2e61963c94a8c73e8a4678fbe89d0ca5d97c525795c48b702f7b5" exitCode=0 Jul 16 16:15:21.150127 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:21.149986 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7447-140072613727376" event={"ID":"911f56bc-7ce7-41b9-9c77-40f60f174134","Type":"ContainerDied","Data":"d371ad1f56f2e61963c94a8c73e8a4678fbe89d0ca5d97c525795c48b702f7b5"} Jul 16 16:15:22.305094 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:22.305070 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:22.329414 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:22.329385 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7447-140072613727376_911f56bc-7ce7-41b9-9c77-40f60f174134/test-curl-7447-140072613727376/0.log" Jul 16 16:15:22.356192 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:22.356154 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:22.358430 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:22.358410 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7447-140072613727376"] Jul 16 16:15:22.395059 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:22.395041 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5xph\" (UniqueName: \"kubernetes.io/projected/911f56bc-7ce7-41b9-9c77-40f60f174134-kube-api-access-d5xph\") pod \"911f56bc-7ce7-41b9-9c77-40f60f174134\" (UID: \"911f56bc-7ce7-41b9-9c77-40f60f174134\") " Jul 16 16:15:22.397055 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:22.397031 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911f56bc-7ce7-41b9-9c77-40f60f174134-kube-api-access-d5xph" (OuterVolumeSpecName: "kube-api-access-d5xph") pod "911f56bc-7ce7-41b9-9c77-40f60f174134" (UID: "911f56bc-7ce7-41b9-9c77-40f60f174134"). InnerVolumeSpecName "kube-api-access-d5xph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:22.496713 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:22.496656 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5xph\" (UniqueName: \"kubernetes.io/projected/911f56bc-7ce7-41b9-9c77-40f60f174134-kube-api-access-d5xph\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:23.161832 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.161802 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7447-140072613727376" Jul 16 16:15:23.162008 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.161809 2573 scope.go:117] "RemoveContainer" containerID="d371ad1f56f2e61963c94a8c73e8a4678fbe89d0ca5d97c525795c48b702f7b5" Jul 16 16:15:23.501697 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.501187 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb"] Jul 16 16:15:23.501697 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.501646 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" podUID="29e13e71-c85c-4f84-861a-9154091eade2" containerName="maas-api" containerID="cri-o://79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f" gracePeriod=30 Jul 16 16:15:23.542271 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.542242 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc"] Jul 16 16:15:23.542450 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.542430 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" podUID="05635f40-5b57-46ac-b73c-a76dc201130f" containerName="cleanup" containerID="cri-o://132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525" gracePeriod=30 Jul 16 16:15:23.768694 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.768662 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:15:23.809082 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.809055 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls\") pod \"29e13e71-c85c-4f84-861a-9154091eade2\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " Jul 16 16:15:23.809232 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.809100 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e13e71-c85c-4f84-861a-9154091eade2-openshift-service-ca\") pod \"29e13e71-c85c-4f84-861a-9154091eade2\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " Jul 16 16:15:23.809232 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.809173 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm5wl\" (UniqueName: \"kubernetes.io/projected/29e13e71-c85c-4f84-861a-9154091eade2-kube-api-access-cm5wl\") pod \"29e13e71-c85c-4f84-861a-9154091eade2\" (UID: \"29e13e71-c85c-4f84-861a-9154091eade2\") " Jul 16 16:15:23.809473 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.809450 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e13e71-c85c-4f84-861a-9154091eade2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29e13e71-c85c-4f84-861a-9154091eade2" (UID: "29e13e71-c85c-4f84-861a-9154091eade2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:15:23.809579 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.809564 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e13e71-c85c-4f84-861a-9154091eade2-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:23.811224 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.811199 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e13e71-c85c-4f84-861a-9154091eade2-kube-api-access-cm5wl" (OuterVolumeSpecName: "kube-api-access-cm5wl") pod "29e13e71-c85c-4f84-861a-9154091eade2" (UID: "29e13e71-c85c-4f84-861a-9154091eade2"). InnerVolumeSpecName "kube-api-access-cm5wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:23.811318 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.811232 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29e13e71-c85c-4f84-861a-9154091eade2" (UID: "29e13e71-c85c-4f84-861a-9154091eade2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:15:23.910047 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.910013 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm5wl\" (UniqueName: \"kubernetes.io/projected/29e13e71-c85c-4f84-861a-9154091eade2-kube-api-access-cm5wl\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:23.910047 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:23.910041 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e13e71-c85c-4f84-861a-9154091eade2-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:24.167910 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.167876 2573 generic.go:356] "Generic (PLEG): container finished" podID="29e13e71-c85c-4f84-861a-9154091eade2" containerID="79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f" exitCode=0 Jul 16 16:15:24.168088 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.167920 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" Jul 16 16:15:24.168088 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.167944 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" event={"ID":"29e13e71-c85c-4f84-861a-9154091eade2","Type":"ContainerDied","Data":"79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f"} Jul 16 16:15:24.168088 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.168007 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb" event={"ID":"29e13e71-c85c-4f84-861a-9154091eade2","Type":"ContainerDied","Data":"610cc552bb950fbe916547dd2ac61e78bd683f5fc98e924d959da000096d2f25"} Jul 16 16:15:24.168088 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.168035 2573 scope.go:117] "RemoveContainer" containerID="79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f" Jul 16 16:15:24.178381 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.178362 2573 scope.go:117] "RemoveContainer" containerID="79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f" Jul 16 16:15:24.178726 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:15:24.178700 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f\": container with ID starting with 79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f not found: ID does not exist" containerID="79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f" Jul 16 16:15:24.178789 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.178740 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f"} err="failed to get container status \"79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f\": rpc error: code = NotFound desc = could not find container \"79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f\": container with ID starting with 79de2052369231a8973e79bfbedae5e7888a8621dec6bcb4c1b43e30b7226f4f not found: ID does not exist" Jul 16 16:15:24.193591 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:15:24.193570 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e13e71_c85c_4f84_861a_9154091eade2.slice/crio-610cc552bb950fbe916547dd2ac61e78bd683f5fc98e924d959da000096d2f25\": RecentStats: unable to find data in memory cache]" Jul 16 16:15:24.202263 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.202243 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb"] Jul 16 16:15:24.209890 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.209868 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-83e7d4-57bc8f7494-6rdtb"] Jul 16 16:15:24.384694 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.384667 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e13e71-c85c-4f84-861a-9154091eade2" path="/var/lib/kubelet/pods/29e13e71-c85c-4f84-861a-9154091eade2/volumes" Jul 16 16:15:24.385195 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:24.385161 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911f56bc-7ce7-41b9-9c77-40f60f174134" path="/var/lib/kubelet/pods/911f56bc-7ce7-41b9-9c77-40f60f174134/volumes" Jul 16 16:15:31.338565 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.338535 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz"] Jul 16 16:15:31.339103 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.338742 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" podUID="9a41b94b-3e05-43b7-b5a5-a182d766d5af" containerName="maas-api" containerID="cri-o://8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64" gracePeriod=30 Jul 16 16:15:31.379884 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.379860 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj"] Jul 16 16:15:31.380071 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.380036 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" podUID="3b609a42-4716-4b02-99ed-f5fd74eb659c" containerName="cleanup" containerID="cri-o://81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76" gracePeriod=30 Jul 16 16:15:31.595997 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.595941 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:15:31.680755 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.680726 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a41b94b-3e05-43b7-b5a5-a182d766d5af-maas-api-tls\") pod \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " Jul 16 16:15:31.680917 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.680819 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41b94b-3e05-43b7-b5a5-a182d766d5af-openshift-service-ca\") pod \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " Jul 16 16:15:31.680917 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.680842 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvggl\" (UniqueName: \"kubernetes.io/projected/9a41b94b-3e05-43b7-b5a5-a182d766d5af-kube-api-access-gvggl\") pod \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\" (UID: \"9a41b94b-3e05-43b7-b5a5-a182d766d5af\") " Jul 16 16:15:31.681203 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.681169 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a41b94b-3e05-43b7-b5a5-a182d766d5af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a41b94b-3e05-43b7-b5a5-a182d766d5af" (UID: "9a41b94b-3e05-43b7-b5a5-a182d766d5af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:15:31.682733 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.682710 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a41b94b-3e05-43b7-b5a5-a182d766d5af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a41b94b-3e05-43b7-b5a5-a182d766d5af" (UID: "9a41b94b-3e05-43b7-b5a5-a182d766d5af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:15:31.682855 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.682832 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a41b94b-3e05-43b7-b5a5-a182d766d5af-kube-api-access-gvggl" (OuterVolumeSpecName: "kube-api-access-gvggl") pod "9a41b94b-3e05-43b7-b5a5-a182d766d5af" (UID: "9a41b94b-3e05-43b7-b5a5-a182d766d5af"). InnerVolumeSpecName "kube-api-access-gvggl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:31.781714 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.781604 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a41b94b-3e05-43b7-b5a5-a182d766d5af-maas-api-tls\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:31.781714 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.781665 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41b94b-3e05-43b7-b5a5-a182d766d5af-openshift-service-ca\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:31.781714 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:31.781691 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvggl\" (UniqueName: \"kubernetes.io/projected/9a41b94b-3e05-43b7-b5a5-a182d766d5af-kube-api-access-gvggl\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:32.215932 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.215901 2573 generic.go:356] "Generic (PLEG): container finished" podID="9a41b94b-3e05-43b7-b5a5-a182d766d5af" containerID="8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64" exitCode=0 Jul 16 16:15:32.216118 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.215944 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" Jul 16 16:15:32.216118 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.215988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" event={"ID":"9a41b94b-3e05-43b7-b5a5-a182d766d5af","Type":"ContainerDied","Data":"8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64"} Jul 16 16:15:32.216118 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.216039 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz" event={"ID":"9a41b94b-3e05-43b7-b5a5-a182d766d5af","Type":"ContainerDied","Data":"5ff7413c5d5a40d0f8f226e534d150a5652dadc57dffe90697f1a990bf2002b3"} Jul 16 16:15:32.216118 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.216061 2573 scope.go:117] "RemoveContainer" containerID="8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64" Jul 16 16:15:32.225193 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.225180 2573 scope.go:117] "RemoveContainer" containerID="8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64" Jul 16 16:15:32.225427 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:15:32.225411 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64\": container with ID starting with 8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64 not found: ID does not exist" containerID="8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64" Jul 16 16:15:32.225471 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.225437 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64"} err="failed to get container status \"8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64\": rpc error: code = NotFound desc = could not find container \"8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64\": container with ID starting with 8e23367199cb752d14afacd32a296ec81eb8a0655cb9d3d0223f60e056a79c64 not found: ID does not exist" Jul 16 16:15:32.238836 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.238811 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz"] Jul 16 16:15:32.242447 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.242426 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-590f9a-76c6f8f554-rpgwz"] Jul 16 16:15:32.384005 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:32.383974 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a41b94b-3e05-43b7-b5a5-a182d766d5af" path="/var/lib/kubelet/pods/9a41b94b-3e05-43b7-b5a5-a182d766d5af/volumes" Jul 16 16:15:38.421865 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:38.421833 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66647745d4-t7vz7_92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61/authorino/0.log" Jul 16 16:15:43.882000 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:43.881970 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-w2b28_1d556ecd-2925-441b-a755-61663894fb5d/manager/0.log" Jul 16 16:15:44.015592 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:44.015559 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-775f5d4dfb-m6fdc_a4eff9f4-e102-42b7-b33f-6f798dadb4e1/manager/0.log" Jul 16 16:15:44.388970 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:44.388944 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd_3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef/copy-manifests/0.log" Jul 16 16:15:44.402759 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:44.402734 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd_3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef/manager/0.log" Jul 16 16:15:45.107192 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:45.107161 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b6d86c5cf-66pgn_57b1e242-4d62-49dc-9a68-7d8aa41cc225/maas-api/0.log" Jul 16 16:15:45.234977 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:45.234953 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29736975-8ccx8_004c035b-5d55-443e-8b23-c0967139e790/cleanup/0.log" Jul 16 16:15:45.354314 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:45.354289 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc_05635f40-5b57-46ac-b73c-a76dc201130f/cleanup/0.log" Jul 16 16:15:45.467359 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:45.467285 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj_3b609a42-4716-4b02-99ed-f5fd74eb659c/cleanup/0.log" Jul 16 16:15:46.222349 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:46.222321 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 16:15:46.229289 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:46.229271 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hprmr_2308a825-c474-4f66-a4ca-12e024550bdf/ovn-acl-logging/0.log" Jul 16 16:15:46.989191 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:46.989163 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66647745d4-t7vz7_92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61/authorino/0.log" Jul 16 16:15:47.676238 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:47.676206 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-gqpkf_848104d7-23a4-404a-8447-1a09e78f714a/manager/0.log" Jul 16 16:15:48.539956 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:48.539931 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bb6d5565c-ccxjn_0f46714d-b3e2-461d-9c6d-5d053e2949a1/kube-auth-proxy/0.log" Jul 16 16:15:53.718343 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:53.718313 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" Jul 16 16:15:53.885035 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:53.884952 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs6jm\" (UniqueName: \"kubernetes.io/projected/05635f40-5b57-46ac-b73c-a76dc201130f-kube-api-access-hs6jm\") pod \"05635f40-5b57-46ac-b73c-a76dc201130f\" (UID: \"05635f40-5b57-46ac-b73c-a76dc201130f\") " Jul 16 16:15:53.887062 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:53.887042 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05635f40-5b57-46ac-b73c-a76dc201130f-kube-api-access-hs6jm" (OuterVolumeSpecName: "kube-api-access-hs6jm") pod "05635f40-5b57-46ac-b73c-a76dc201130f" (UID: "05635f40-5b57-46ac-b73c-a76dc201130f"). InnerVolumeSpecName "kube-api-access-hs6jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:15:53.986012 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:53.985990 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs6jm\" (UniqueName: \"kubernetes.io/projected/05635f40-5b57-46ac-b73c-a76dc201130f-kube-api-access-hs6jm\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:15:54.165578 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165504 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dnqzv/must-gather-tjn5t"] Jul 16 16:15:54.165933 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165920 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911f56bc-7ce7-41b9-9c77-40f60f174134" containerName="test-curl-7447-140072613727376" Jul 16 16:15:54.165978 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165935 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="911f56bc-7ce7-41b9-9c77-40f60f174134" containerName="test-curl-7447-140072613727376" Jul 16 16:15:54.165978 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165949 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05635f40-5b57-46ac-b73c-a76dc201130f" containerName="cleanup" Jul 16 16:15:54.165978 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165954 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="05635f40-5b57-46ac-b73c-a76dc201130f" containerName="cleanup" Jul 16 16:15:54.165978 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165967 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e13e71-c85c-4f84-861a-9154091eade2" containerName="maas-api" Jul 16 16:15:54.165978 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165974 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e13e71-c85c-4f84-861a-9154091eade2" containerName="maas-api" Jul 16 16:15:54.166132 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165982 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a41b94b-3e05-43b7-b5a5-a182d766d5af" containerName="maas-api" Jul 16 16:15:54.166132 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.165988 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a41b94b-3e05-43b7-b5a5-a182d766d5af" containerName="maas-api" Jul 16 16:15:54.166132 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.166067 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="05635f40-5b57-46ac-b73c-a76dc201130f" containerName="cleanup" Jul 16 16:15:54.166132 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.166075 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="911f56bc-7ce7-41b9-9c77-40f60f174134" containerName="test-curl-7447-140072613727376" Jul 16 16:15:54.166132 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.166082 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a41b94b-3e05-43b7-b5a5-a182d766d5af" containerName="maas-api" Jul 16 16:15:54.166132 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.166088 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e13e71-c85c-4f84-861a-9154091eade2" containerName="maas-api" Jul 16 16:15:54.169402 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.169381 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnqzv/must-gather-tjn5t" Jul 16 16:15:54.172330 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.172310 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnqzv\"/\"kube-root-ca.crt\"" Jul 16 16:15:54.172420 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.172360 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dnqzv\"/\"default-dockercfg-5jg58\"" Jul 16 16:15:54.173419 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.173404 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnqzv\"/\"openshift-service-ca.crt\"" Jul 16 16:15:54.178061 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.178042 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnqzv/must-gather-tjn5t"] Jul 16 16:15:54.289519 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.289486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9-must-gather-output\") pod \"must-gather-tjn5t\" (UID: \"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9\") " pod="openshift-must-gather-dnqzv/must-gather-tjn5t" Jul 16 16:15:54.289709 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.289529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5pn\" (UniqueName: \"kubernetes.io/projected/a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9-kube-api-access-2f5pn\") pod \"must-gather-tjn5t\" (UID: \"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9\") " pod="openshift-must-gather-dnqzv/must-gather-tjn5t" Jul 16 16:15:54.329834 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.329806 2573 generic.go:356] "Generic (PLEG): container finished" podID="05635f40-5b57-46ac-b73c-a76dc201130f" containerID="132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525" exitCode=137 Jul 16 16:15:54.329998 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.329858 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" Jul 16 16:15:54.329998 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.329886 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" event={"ID":"05635f40-5b57-46ac-b73c-a76dc201130f","Type":"ContainerDied","Data":"132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525"} Jul 16 16:15:54.329998 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.329928 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc" event={"ID":"05635f40-5b57-46ac-b73c-a76dc201130f","Type":"ContainerDied","Data":"7dc1a55902dad40c3fd8aabd7dae1e281ea0c1e19952bcc6863de4820c897f86"} Jul 16 16:15:54.329998 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.329950 2573 scope.go:117] "RemoveContainer" containerID="132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525" Jul 16 16:15:54.338997 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.338982 2573 scope.go:117] "RemoveContainer" containerID="132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525" Jul 16 16:15:54.339234 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:15:54.339214 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525\": container with ID starting with 132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525 not found: ID does not exist" containerID="132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525" Jul 16 16:15:54.339291 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.339243 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525"} err="failed to get container status \"132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525\": rpc error: code = NotFound desc = could not find container \"132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525\": container with ID starting with 132ae80b3b5963509bd36f497940ae763117dfff5d484747a9f0536fa4e3c525 not found: ID does not exist" Jul 16 16:15:54.353690 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.353659 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc"] Jul 16 16:15:54.361774 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.361754 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-83e7d4-29736975-w5rdc"] Jul 16 16:15:54.383785 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.383762 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05635f40-5b57-46ac-b73c-a76dc201130f" path="/var/lib/kubelet/pods/05635f40-5b57-46ac-b73c-a76dc201130f/volumes" Jul 16 16:15:54.390091 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.390075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9-must-gather-output\") pod \"must-gather-tjn5t\" (UID: \"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9\") " pod="openshift-must-gather-dnqzv/must-gather-tjn5t" Jul 16 16:15:54.390150 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.390118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5pn\" (UniqueName: \"kubernetes.io/projected/a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9-kube-api-access-2f5pn\") pod \"must-gather-tjn5t\" (UID: \"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9\") " pod="openshift-must-gather-dnqzv/must-gather-tjn5t" Jul 16 16:15:54.390373 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.390356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9-must-gather-output\") pod \"must-gather-tjn5t\" (UID: \"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9\") " pod="openshift-must-gather-dnqzv/must-gather-tjn5t" Jul 16 16:15:54.398966 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.398941 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5pn\" (UniqueName: \"kubernetes.io/projected/a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9-kube-api-access-2f5pn\") pod \"must-gather-tjn5t\" (UID: \"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9\") " pod="openshift-must-gather-dnqzv/must-gather-tjn5t" Jul 16 16:15:54.479967 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.479917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnqzv/must-gather-tjn5t" Jul 16 16:15:54.613279 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:54.613252 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnqzv/must-gather-tjn5t"] Jul 16 16:15:54.614905 ip-10-0-128-48 kubenswrapper[2573]: W0716 16:15:54.614877 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda012d7bf_f2d6_4b4d_b9c0_1796b1ab13a9.slice/crio-6a3bfad1466cb791e888387bc29ec16946d632b3b590528529555a987c7738bc WatchSource:0}: Error finding container 6a3bfad1466cb791e888387bc29ec16946d632b3b590528529555a987c7738bc: Status 404 returned error can't find the container with id 6a3bfad1466cb791e888387bc29ec16946d632b3b590528529555a987c7738bc Jul 16 16:15:55.336877 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:55.336838 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnqzv/must-gather-tjn5t" event={"ID":"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9","Type":"ContainerStarted","Data":"6a3bfad1466cb791e888387bc29ec16946d632b3b590528529555a987c7738bc"} Jul 16 16:15:56.347481 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:56.347449 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnqzv/must-gather-tjn5t" event={"ID":"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9","Type":"ContainerStarted","Data":"b8bba441a3a3d978e27621d57b674fe5364c7fa0e421022b237a6040185df810"} Jul 16 16:15:56.347481 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:56.347488 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnqzv/must-gather-tjn5t" event={"ID":"a012d7bf-f2d6-4b4d-b9c0-1796b1ab13a9","Type":"ContainerStarted","Data":"656e046fad428a47c3cb5457f43e37bcd79b5e4b7e2bb1447dd2d7751890309e"} Jul 16 16:15:56.367203 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:56.367135 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dnqzv/must-gather-tjn5t" podStartSLOduration=1.535552085 podStartE2EDuration="2.367112472s" totalImagesPullingTime="831.560387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:15:56.364832897 +0000 UTC m=+2410.803359722" watchObservedRunningTime="2026-07-16 16:15:56.367112472 +0000 UTC m=+2410.805639287" Jul 16 16:15:57.196737 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:57.196709 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sdfct_e7962491-319d-4d1c-8d70-74706e135117/global-pull-secret-syncer/0.log" Jul 16 16:15:57.339535 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:57.339505 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qxxr2_065bbaff-e023-4be5-bd5b-be46329be9a3/konnectivity-agent/0.log" Jul 16 16:15:57.365903 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:15:57.365879 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-48.ec2.internal_fc266b970294ca7e6b02e58f7dfbfaf7/haproxy/0.log" Jul 16 16:16:00.012671 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:00.012630 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8"] Jul 16 16:16:00.013477 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:00.012899 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736975-8ccx8" podUID="004c035b-5d55-443e-8b23-c0967139e790" containerName="cleanup" containerID="cri-o://df6f3d55a457828f5a7ac66f4766d2063254aec44183dae9c83b60653894ee63" gracePeriod=30 Jul 16 16:16:01.659096 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:01.659013 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" Jul 16 16:16:01.667045 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:01.667007 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54tdj\" (UniqueName: \"kubernetes.io/projected/3b609a42-4716-4b02-99ed-f5fd74eb659c-kube-api-access-54tdj\") pod \"3b609a42-4716-4b02-99ed-f5fd74eb659c\" (UID: \"3b609a42-4716-4b02-99ed-f5fd74eb659c\") " Jul 16 16:16:01.674786 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:01.674735 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b609a42-4716-4b02-99ed-f5fd74eb659c-kube-api-access-54tdj" (OuterVolumeSpecName: "kube-api-access-54tdj") pod "3b609a42-4716-4b02-99ed-f5fd74eb659c" (UID: "3b609a42-4716-4b02-99ed-f5fd74eb659c"). InnerVolumeSpecName "kube-api-access-54tdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:16:01.771068 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:01.771018 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54tdj\" (UniqueName: \"kubernetes.io/projected/3b609a42-4716-4b02-99ed-f5fd74eb659c-kube-api-access-54tdj\") on node \"ip-10-0-128-48.ec2.internal\" DevicePath \"\"" Jul 16 16:16:02.419198 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.417628 2573 generic.go:356] "Generic (PLEG): container finished" podID="3b609a42-4716-4b02-99ed-f5fd74eb659c" containerID="81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76" exitCode=137 Jul 16 16:16:02.419198 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.417858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" event={"ID":"3b609a42-4716-4b02-99ed-f5fd74eb659c","Type":"ContainerDied","Data":"81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76"} Jul 16 16:16:02.419198 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.417925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" event={"ID":"3b609a42-4716-4b02-99ed-f5fd74eb659c","Type":"ContainerDied","Data":"12517eaedbfc7612efce9f4c2373ff55e08d4796d4ed1ab38644c3ec38211006"} Jul 16 16:16:02.419198 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.417964 2573 scope.go:117] "RemoveContainer" containerID="81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76" Jul 16 16:16:02.419198 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.418143 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj" Jul 16 16:16:02.437452 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.437056 2573 scope.go:117] "RemoveContainer" containerID="81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76" Jul 16 16:16:02.437830 ip-10-0-128-48 kubenswrapper[2573]: E0716 16:16:02.437794 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76\": container with ID starting with 81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76 not found: ID does not exist" containerID="81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76" Jul 16 16:16:02.437971 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.437831 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76"} err="failed to get container status \"81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76\": rpc error: code = NotFound desc = could not find container \"81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76\": container with ID starting with 81c6bc2ac72e848e4a5564400af25d6964edadd4f471e195c4d9691df69c4f76 not found: ID does not exist" Jul 16 16:16:02.456426 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.455876 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj"] Jul 16 16:16:02.457238 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.457213 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-590f9a-29736975-z9psj"] Jul 16 16:16:02.473078 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.473030 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66647745d4-t7vz7_92a7ebd8-bcbf-4ed7-8716-4c5bd694dd61/authorino/0.log" Jul 16 16:16:02.811075 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:02.810719 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-gqpkf_848104d7-23a4-404a-8447-1a09e78f714a/manager/0.log" Jul 16 16:16:04.387807 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:04.387775 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b609a42-4716-4b02-99ed-f5fd74eb659c" path="/var/lib/kubelet/pods/3b609a42-4716-4b02-99ed-f5fd74eb659c/volumes" Jul 16 16:16:04.642351 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:04.642261 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5cd44bf4-cc3c-461b-ab91-825f7adffdaa/alertmanager/0.log" Jul 16 16:16:04.676436 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:04.676410 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5cd44bf4-cc3c-461b-ab91-825f7adffdaa/config-reloader/0.log" Jul 16 16:16:04.737152 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:04.737124 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5cd44bf4-cc3c-461b-ab91-825f7adffdaa/kube-rbac-proxy-web/0.log" Jul 16 16:16:04.805437 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:04.805373 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5cd44bf4-cc3c-461b-ab91-825f7adffdaa/kube-rbac-proxy/0.log" Jul 16 16:16:04.873316 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:04.873285 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5cd44bf4-cc3c-461b-ab91-825f7adffdaa/kube-rbac-proxy-metric/0.log" Jul 16 16:16:04.931666 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:04.931566 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5cd44bf4-cc3c-461b-ab91-825f7adffdaa/prom-label-proxy/0.log" Jul 16 16:16:05.004056 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.004028 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5cd44bf4-cc3c-461b-ab91-825f7adffdaa/init-config-reloader/0.log" Jul 16 16:16:05.116664 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.116633 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-qzr2s_4f12b5ca-d7d4-4042-bf06-fe8b14625a8a/kube-state-metrics/0.log" Jul 16 16:16:05.141782 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.141750 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-qzr2s_4f12b5ca-d7d4-4042-bf06-fe8b14625a8a/kube-rbac-proxy-main/0.log" Jul 16 16:16:05.173824 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.173797 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-qzr2s_4f12b5ca-d7d4-4042-bf06-fe8b14625a8a/kube-rbac-proxy-self/0.log" Jul 16 16:16:05.213273 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.213189 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8468d64cff-92r2h_f6a9b491-69ae-4979-a4f7-7f62bd49043b/metrics-server/0.log" Jul 16 16:16:05.243493 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.243469 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-svnwp_4eaa8f7c-4a05-4758-85f7-88d84d55b7af/monitoring-plugin/0.log" Jul 16 16:16:05.272179 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.272154 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-786zz_9b02f526-ec6f-4404-96fe-f98f1a013a54/node-exporter/0.log" Jul 16 16:16:05.296150 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.296123 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-786zz_9b02f526-ec6f-4404-96fe-f98f1a013a54/kube-rbac-proxy/0.log" Jul 16 16:16:05.328629 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.328585 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-786zz_9b02f526-ec6f-4404-96fe-f98f1a013a54/init-textfile/0.log" Jul 16 16:16:05.531758 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.531730 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-kjz42_32bb372b-72c8-416e-b667-d6e8e1b3d9fb/kube-rbac-proxy-main/0.log" Jul 16 16:16:05.555287 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.555252 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-kjz42_32bb372b-72c8-416e-b667-d6e8e1b3d9fb/kube-rbac-proxy-self/0.log" Jul 16 16:16:05.579037 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.579012 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-kjz42_32bb372b-72c8-416e-b667-d6e8e1b3d9fb/openshift-state-metrics/0.log" Jul 16 16:16:05.619852 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.619820 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b918f262-9202-4a35-8f85-169a7d68f627/prometheus/0.log" Jul 16 16:16:05.648164 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.648136 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b918f262-9202-4a35-8f85-169a7d68f627/config-reloader/0.log" Jul 16 16:16:05.675443 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.675426 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b918f262-9202-4a35-8f85-169a7d68f627/thanos-sidecar/0.log" Jul 16 16:16:05.698686 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.698656 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b918f262-9202-4a35-8f85-169a7d68f627/kube-rbac-proxy-web/0.log" Jul 16 16:16:05.727443 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.727036 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b918f262-9202-4a35-8f85-169a7d68f627/kube-rbac-proxy/0.log" Jul 16 16:16:05.752484 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.752450 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b918f262-9202-4a35-8f85-169a7d68f627/kube-rbac-proxy-thanos/0.log" Jul 16 16:16:05.781864 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.781789 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b918f262-9202-4a35-8f85-169a7d68f627/init-config-reloader/0.log" Jul 16 16:16:05.815751 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.815691 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-jslpn_4f399840-a744-424e-88b5-43feb12b4f8f/prometheus-operator/0.log" Jul 16 16:16:05.840739 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.840713 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-jslpn_4f399840-a744-424e-88b5-43feb12b4f8f/kube-rbac-proxy/0.log" Jul 16 16:16:05.906225 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.906200 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7554f75ffc-dckgm_656dcbdb-23de-4e81-a62f-49552fb6df29/telemeter-client/0.log" Jul 16 16:16:05.930518 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.930491 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7554f75ffc-dckgm_656dcbdb-23de-4e81-a62f-49552fb6df29/reload/0.log" Jul 16 16:16:05.954099 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.954072 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7554f75ffc-dckgm_656dcbdb-23de-4e81-a62f-49552fb6df29/kube-rbac-proxy/0.log" Jul 16 16:16:05.981595 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:05.981563 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d844dddbc-x58qz_3346f6e5-0fbd-4171-a070-bbd198f9d4f9/thanos-query/0.log" Jul 16 16:16:06.004602 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.004571 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d844dddbc-x58qz_3346f6e5-0fbd-4171-a070-bbd198f9d4f9/kube-rbac-proxy-web/0.log" Jul 16 16:16:06.028148 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.028124 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d844dddbc-x58qz_3346f6e5-0fbd-4171-a070-bbd198f9d4f9/kube-rbac-proxy/0.log" Jul 16 16:16:06.051991 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.051915 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d844dddbc-x58qz_3346f6e5-0fbd-4171-a070-bbd198f9d4f9/prom-label-proxy/0.log" Jul 16 16:16:06.073307 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.073269 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d844dddbc-x58qz_3346f6e5-0fbd-4171-a070-bbd198f9d4f9/kube-rbac-proxy-rules/0.log" Jul 16 16:16:06.095177 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.095153 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d844dddbc-x58qz_3346f6e5-0fbd-4171-a070-bbd198f9d4f9/kube-rbac-proxy-metrics/0.log" Jul 16 16:16:06.686189 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.686152 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb"] Jul 16 16:16:06.686784 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.686762 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b609a42-4716-4b02-99ed-f5fd74eb659c" containerName="cleanup" Jul 16 16:16:06.686857 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.686786 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b609a42-4716-4b02-99ed-f5fd74eb659c" containerName="cleanup" Jul 16 16:16:06.686916 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.686907 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b609a42-4716-4b02-99ed-f5fd74eb659c" containerName="cleanup" Jul 16 16:16:06.691226 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.691202 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.702098 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.702071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb"] Jul 16 16:16:06.778282 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.778230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xsqj\" (UniqueName: \"kubernetes.io/projected/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-kube-api-access-7xsqj\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.778587 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.778565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-proc\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.778768 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.778741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-podres\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.778944 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.778909 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-sys\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.779123 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.779090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-lib-modules\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.879970 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.879927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xsqj\" (UniqueName: \"kubernetes.io/projected/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-kube-api-access-7xsqj\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.880153 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.879994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-proc\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.880153 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.880035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-podres\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.880153 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.880076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-sys\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.880153 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.880125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-lib-modules\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.880378 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.880147 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-proc\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.880378 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.880294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-lib-modules\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.880378 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.880335 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-sys\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.880378 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.880351 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-podres\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:06.889690 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:06.889656 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xsqj\" (UniqueName: \"kubernetes.io/projected/fc069111-f875-499a-b0e4-d8fd8f7cdd9a-kube-api-access-7xsqj\") pod \"perf-node-gather-daemonset-sb9mb\" (UID: \"fc069111-f875-499a-b0e4-d8fd8f7cdd9a\") " pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:07.005722 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:07.005608 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:07.192505 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:07.192431 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb"] Jul 16 16:16:07.484913 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:07.484878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" event={"ID":"fc069111-f875-499a-b0e4-d8fd8f7cdd9a","Type":"ContainerStarted","Data":"ca0968fb9b5ade4674109c060edb4c467d793548addbc0dadf76e55aa4f7b3c3"} Jul 16 16:16:07.485086 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:07.484929 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" event={"ID":"fc069111-f875-499a-b0e4-d8fd8f7cdd9a","Type":"ContainerStarted","Data":"027e25954da1e4b75a29d5330bf05e836d7a455afc5197a0f9d60c41a072c328"} Jul 16 16:16:07.485086 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:07.484971 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:07.509223 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:07.509174 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" podStartSLOduration=1.509152555 podStartE2EDuration="1.509152555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:16:07.507267822 +0000 UTC m=+2421.945794644" watchObservedRunningTime="2026-07-16 16:16:07.509152555 +0000 UTC m=+2421.947679379" Jul 16 16:16:08.445312 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:08.445271 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b6767c8c5-xvk2l_0e453bb7-307f-4733-870d-f1035340b3b9/console/0.log" Jul 16 16:16:10.321904 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:10.321844 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tr9ts_0e5984f0-0dcb-467c-876d-84c7943edc28/dns/0.log" Jul 16 16:16:10.364799 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:10.364770 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tr9ts_0e5984f0-0dcb-467c-876d-84c7943edc28/kube-rbac-proxy/0.log" Jul 16 16:16:10.410027 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:10.410001 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dh7cp_9295763d-a0e9-43f3-98c9-a86588ddf8f2/dns-node-resolver/0.log" Jul 16 16:16:11.147545 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:11.147466 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6s8bz_cbf6f6e9-5c8d-4aa1-9d9a-dab9a39240d4/node-ca/0.log" Jul 16 16:16:12.318406 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:12.318371 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bb6d5565c-ccxjn_0f46714d-b3e2-461d-9c6d-5d053e2949a1/kube-auth-proxy/0.log" Jul 16 16:16:13.362513 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:13.362480 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t99dx_1a58c255-8ae3-46f3-977e-310e0db54caa/serve-healthcheck-canary/0.log" Jul 16 16:16:13.505096 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:13.504984 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dnqzv/perf-node-gather-daemonset-sb9mb" Jul 16 16:16:14.161811 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:14.161786 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfqqs_5065548b-266f-4194-b8fe-f44fb431cdd4/kube-rbac-proxy/0.log" Jul 16 16:16:14.198499 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:14.198467 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfqqs_5065548b-266f-4194-b8fe-f44fb431cdd4/exporter/0.log" Jul 16 16:16:14.226900 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:14.226876 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfqqs_5065548b-266f-4194-b8fe-f44fb431cdd4/extractor/0.log" Jul 16 16:16:16.931066 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:16.930990 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-w2b28_1d556ecd-2925-441b-a755-61663894fb5d/manager/0.log" Jul 16 16:16:17.046813 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:17.046779 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-775f5d4dfb-m6fdc_a4eff9f4-e102-42b7-b33f-6f798dadb4e1/manager/0.log" Jul 16 16:16:17.452541 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:17.452506 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd_3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef/manager/0.log" Jul 16 16:16:17.560878 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:17.560787 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b5b94d4fb-cdrqd_3ed48ac9-6e9a-40d1-adf9-3dae3c5683ef/copy-manifests/0.log" Jul 16 16:16:17.939655 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:17.939608 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bb6d5565c-ccxjn_0f46714d-b3e2-461d-9c6d-5d053e2949a1/kube-auth-proxy/0.log" Jul 16 16:16:18.991887 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:18.991801 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-75498485d8-l6dnk_b8cdb65b-8aab-4819-85a2-64795faaa86d/manager/0.log" Jul 16 16:16:19.096222 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:19.096186 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ggnfk_6b29c0bf-6034-43e7-b04c-1dd7156bfe72/openshift-lws-operator/0.log" Jul 16 16:16:19.476327 ip-10-0-128-48 kubenswrapper[2573]: I0716 16:16:19.476303 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bb6d5565c-ccxjn_0f46714d-b3e2-461d-9c6d-5d053e2949a1/kube-auth-proxy/0.log"